Image processing device and method, and recording medium

ABSTRACT

An effective pixel area calculating circuit ( 11 ) detects position information indicating the position of a target pixel in a frame. A lacking pixel creating circuit ( 12 ) determines the class of the target pixel from a plurality of classes in accordance with the position information, then selects a plurality of pixels from an input image signal as a prediction tap, and carries out arithmetic processing based on conversion data obtained in advance by learning for each class and the prediction tap, thus outputting an output image signal of higher quality than the input image signal.

TECHNICAL FIELD

This invention relates to an image processing device and method and arecording medium, and particularly to an image processing device andmethod and a recording medium for processing or converting images.

BACKGROUND ART

As a technique for improving the quality such as the resolution of animage signal, for example, classification adaptive processing isemployed, which is disclosed in the Japanese Publication of UnexaminedPatent Application No. H9-74543 and in the specification of thecorresponding U.S. Pat. No. 5,946,044.

In classification adaptive processing, a class tap and a prediction tapfor each target pixel of an input image signal are obtained from theinput image signal. The target pixel is classified into one of presetclasses on the basis of the class tap, and arithmetic operation iscarried out using a prediction coefficient set generated in advance bylearning for each class, selected correspondingly to the classification,and the prediction tap. Thus, an output image signal having higherquality than the input image signal is generated.

In classification adaptive processing, the class tap and the predictiontap of the pixel might be situated outside the effective range of theimage. In this case, there is a high possibility that the pixel outsidethe effective range does not have a normal pixel value. Therefore, inthe conventional classification adaptive processing, the pixel with theclass tap and the prediction tap situated outside the effective range ofthe image is masked as shown in FIG. 1 and thus is not used.

As an example of this classification adaptive processing, a lackingpixel due to failure to correct by an error correcting code or due topacket loss is re-created by classification adaptive processing usingthe surrounding pixels as a class tap and a prediction tap.

In this case, too, as shown in FIG. 1, the pixel value can be set byclassification adaptive processing using the pixel values of the pixelssituated around the target pixel.

Conventionally, processing with so-called the same tap structure iscarried out using pixels which have relatively similar positionalrelations with the lacking pixel in the whole screen.

If the pixels situated around the lacking pixel are outside theeffective range of the image, the set pixel value is not a normal value.Therefore, the pixels situated on the edges of the resultant image aremasked as shown in FIG. 1 and thus are not used.

Moreover, conventionally, similar processing is carried out regardlessof the physical positions of pixels on the screen.

When the image is masked, the image is reduced in size and theresolution of the resultant image is substantially lowered. Moreover,since the processing contents are not changed in accordance with thepositions of pixels on the screen and similar processing is carried outregardless of the physical positions of pixels on the screen,significant improvement in the quality is not realized.

DISCLOSURE OF THE INVENTION

In view of the foregoing status of the art, it is an object of thepresent invention to enable constant generation of an image of highquality regardless of the positions of pixels on the screen.

An image processing device according to the present invention comprises:position detecting means for detecting position information indicatingthe position within a frame, of a target pixel of an input image signalconsisting of a plurality of pixels; class determining means fordetermining the class of the target pixel from a plurality of classes inaccordance with the position information; prediction tap selecting meansfor selecting a plurality of pixels from the input image signal as aprediction tap; and operation means for carrying out arithmeticprocessing based on conversion data obtained in advance by learning foreach of the classes and the prediction tap, thus outputting an outputimage signal of higher quality than the input image signal.

An image processing method according to the present invention comprises:a position detecting step of detecting position information indicatingthe position within a frame, of a target pixel of an input image signalconsisting of a plurality of pixels; a class determining step ofdetermining the class of the target pixel from a plurality of classes inaccordance with the position information; a prediction tap selectingstep of selecting a plurality of pixels from the input image signal as aprediction tap; and an operation step of carrying out arithmeticprocessing based on conversion data obtained in advance by learning foreach of the classes and the prediction tap, thus outputting an outputimage signal of higher quality than the input image signal.

A recording medium according to the present invention has recordedthereon a program for causing a computer to execute image processing,the program comprising; a position detecting step of detecting positioninformation indicating the position within a frame, of a target pixel ofan input image signal consisting of a plurality of pixels; a classdetermining step of determining the class of the target pixel from aplurality of classes in accordance with the position information; aprediction tap selecting step of selecting a plurality of pixels fromthe input image signal as a prediction tap; and an operation step ofcarrying out arithmetic processing based on conversion data obtained inadvance by learning for each of the classes and the prediction tap, thusoutputting an output image signal of higher quality than the input imagesignal.

An image processing device according to the present invention comprises:position detecting means for detecting position information indicatingthe position within a frame, of a target pixel of an input image signalconsisting of a plurality of pixels; class tap selecting means forselecting a plurality of pixels having their positional relations withthe target pixel varied in accordance with the position information,from the input image signal, as a class tap; class determining means fordetermining the class of the target pixel from a plurality of classes inaccordance with the class tap; prediction tap selecting means forselecting a plurality of pixels from the input image signal as aprediction tap; and operation means for carrying out arithmeticprocessing based on conversion data obtained in advance by learning foreach of the classes and the prediction tap, thus outputting an outputimage signal of higher quality than the input image signal.

An image processing method according to the present invention comprises:a position detecting step of detecting position information indicatingthe position within a frame, of a target pixel of an input image signalconsisting of a plurality of pixels; a class tap selecting step ofselecting a plurality of pixels having their positional relations withthe target pixel varied in accordance with the position information,from the input image signal, as a class tap; a class determining step ofdetermining the class of the target pixel from a plurality of classes inaccordance with the class tap; a prediction tap selecting step ofselecting a plurality of pixels from the input image signal as aprediction tap; and an operation step of carrying out arithmeticprocessing based on conversion data obtained in advance by learning foreach of the classes and the prediction tap, thus outputting an outputimage signal of higher quality than the input image signal.

A recording medium according to the present invention has recordedthereon a program for causing a computer to execute image processing,the program comprising: a position detecting step of detecting positioninformation indicating the position within a frame, of a target pixel ofan input image signal consisting of a plurality of pixels; a class tapselecting step of selecting a plurality of pixels having theirpositional relations with the target pixel varied in accordance with theposition information, from the input image signal, as a class tap; aclass determining step of determining the class of the target pixel froma plurality of classes in accordance with the class tap; a predictiontap selecting step of selecting a plurality of pixels from the inputimage signal as a prediction tap; and an operation step of carrying outarithmetic processing based on conversion data obtained in advance bylearning for each of the classes and the prediction tap, thus outputtingan output image signal of higher quality than the input image signal.

An image processing device according to the present invention comprises:position detecting means for detecting position information indicatingthe position within a frame, of a target pixel of an input image signalconsisting of a plurality of pixels; class tap selecting means forselecting a plurality of pixels from the input image signal as a classtap; class determining means for determining the class of the targetpixel from a plurality of classes in accordance with the class tap;prediction tap selecting means for selecting a plurality of pixelshaving their positional relations with the target pixel varied inaccordance with the position information, from the input image signal asa prediction tap; and operation means for carrying out arithmeticprocessing based on conversion data obtained in advance by learning foreach of the classes and the prediction tap, thus outputting an outputimage signal of higher quality than the input image signal.

An image processing method according to the present invention comprises:a position detecting step of detecting position information indicatingthe position within a frame, of a target pixel of an input image signalconsisting of a plurality of pixels; a class tap selecting step ofselecting a plurality of pixels from the input image signal as a classtap; a class determining step of determining the class of the targetpixel from a plurality of classes in accordance with the class tap; aprediction tap selecting step of selecting a plurality of pixels havingtheir positional relations with the target pixel varied in accordancewith the position information, from the input image signal as aprediction tap; and an operation step of carrying out arithmeticprocessing based on conversion data obtained in advance by learning foreach of the classes and the prediction tap, thus outputting an outputimage signal of higher quality than the input image signal.

A recording medium according to the present invention has recordedthereon a program for causing a computer to execute image processing,the program comprising: a position detecting step of detecting positioninformation indicating the position within a frame, of a target pixel ofan input image signal consisting of a plurality of pixels; a class tapselecting step of selecting a plurality of pixels from the input imagesignal as a class tap; a class determining step of determining the classof the target pixel from a plurality of classes in accordance with theclass tap; a prediction tap selecting step of selecting a plurality ofpixels having their positional relations with the target pixel varied inaccordance with the position information, from the input image signal asa prediction tap; and an operation step of carrying out arithmeticprocessing based on conversion data obtained in advance by learning foreach of the classes and the prediction tap, thus outputting an outputimage signal of higher quality than the input image signal.

An image processing device according to the present invention comprises:provisional class tap selecting means for selecting a plurality ofpixels from an input image signal as a provisional class tap, for eachtarget pixel of the input image signal consisting of a plurality ofpixels; true class tap selecting means for selecting a plurality ofpixels having their positional relations with the target pixel varied inaccordance with the position of the provisional class tap within aframe, from the input image signal, as a true class tap; classdetermining means for determining the class of the target pixel from aplurality of classes on the basis of the true class tap; prediction tapselecting means for selecting a plurality of pixels from the input imagesignal as a prediction tap; and operation means for carrying outarithmetic processing based on conversion data obtained in advance bylearning for each of the classes and the prediction tap, thus outputtingan output image signal of higher quality than the input image signal.

An image processing method according to the present invention comprises:a provisional class tap selecting step of selecting a plurality ofpixels from an input image signal as a provisional class tap, for eachtarget pixel of the input image signal consisting of a plurality ofpixels; a true class tap selecting step of selecting a plurality ofpixels having their positional relations with the target pixel varied inaccordance with the position of the provisional class tap within aframe, from the input image signal, as a true class tap; a classdetermining step of determining the class of the target pixel from aplurality of classes on the basis of the true class tap; a predictiontap selecting step of selecting a plurality of pixels from the inputimage signal as a prediction tap; and an operation step of carrying outarithmetic processing based on conversion data obtained in advance bylearning for each of the classes and the prediction tap, thus outputtingan output image signal of higher quality than the input image signal.

A recording medium according to the present invention has recordedthereon a program for causing a computer to execute image processing,the program comprising: a provisional class tap selecting step ofselecting a plurality of pixels from an input image signal as aprovisional class tap, for each target pixel of the input image signalconsisting of a plurality of pixels; a true class tap selecting step ofselecting a plurality of pixels having their positional relations withthe target pixel varied in accordance with the position of theprovisional class tap within a frame, from the input image signal, as atrue class tap; a class determining step of determining the class of thetarget pixel from a plurality of classes on the basis of the true classtap; a prediction tap selecting step of selecting a plurality of pixelsfrom the input image signal as a prediction tap; and an operation stepof carrying out arithmetic processing based on conversion data obtainedin advance by learning for each of the classes and the prediction tap,thus outputting an output image signal of higher quality than the inputimage signal.

An image processing device according to the present invention comprises:class tap selecting means for selecting a plurality of pixels from aninput image signal as a class tap, for each target pixel of the inputimage signal consisting of a plurality of pixels; class determiningmeans for determining the class of the target pixel from a plurality ofclasses on the basis of the class tap; provisional prediction tapselecting means for selecting a plurality of pixels for said each targetpixel from the input image signal as a provisional prediction tap; trueprediction tap selecting means for selecting a plurality of pixelshaving their positional relations with the target pixel varied inaccordance with the position of the provisional prediction tap within aframe, from the input image signal as a true prediction tap; andoperation means for carrying out arithmetic processing based onconversion data obtained in advance by learning for each of the classesand the true prediction tap, thus outputting an output image signal ofhigher quality than the input image signal.

An image processing method according to the present invention comprises:a class tap selecting step selecting a plurality of pixels from an inputimage signal as a class tap, for each target pixel of the input imagesignal consisting of a plurality of pixels; a class determining step ofdetermining the class of the target pixel from a plurality of classes onthe basis of the class tap; a provisional prediction tap selecting stepof selecting a plurality of pixels for said each target pixel from theinput image signal as a provisional prediction tap; a true predictiontap selecting step of selecting a plurality of pixels having theirpositional relations with the target pixel varied in accordance with theposition of the provisional prediction tap within a frame, from theinput image signal as a true prediction tap; and an operation step ofcarrying out arithmetic processing based on conversion data obtained inadvance by learning for each of the classes and the true prediction tap,thus outputting an output image signal of higher quality than the inputimage signal.

A recording medium according to the present invention has recordedthereon a program for causing a computer to execute image processing,the program comprising: a class tap selecting step selecting a pluralityof pixels from an input image signal as a class tap, for each targetpixel of the input image signal consisting of a plurality of pixels; aclass determining step of determining the class of the target pixel froma plurality of classes on the basis of the class tap; a provisionalprediction tap selecting step of selecting a plurality of pixels forsaid each target pixel from the input image signal as a provisionalprediction tap; a true prediction tap selecting step of selecting aplurality of pixels having their positional relations with the targetpixel varied in accordance with the position of the provisionalprediction tap within a frame, from the input image signal as a trueprediction tap; and an operation step of carrying out arithmeticprocessing based on conversion data obtained in advance by learning foreach of the classes and the true prediction tap, thus outputting anoutput image signal of higher quality than the input image signal.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a mask of pixels.

FIG. 2 shows the structure of an embodiment of an image processingdevice according to the present invention.

FIG. 3 shows an exemplary structure of an effective pixel areacalculating circuit.

FIG. 4 illustrates an effective pixel area vertical flag VF and aneffective pixel area horizontal flag HF.

FIG. 5 illustrates pixels around a target of creation.

FIG. 6 shows an exemplary construction of a tap at an edge of an image.

FIG. 7 shows an exemplary construction of a tap at an edge of an image.

FIG. 8 is a block diagram showing the structure of a lacking pixelcreating circuit.

FIG. 9 is a flowchart for explaining the processing at a preprocessingcircuit.

FIG. 10 shows the structure of a motion class generating circuit.

FIG. 11 shows the structure of a motion detecting circuit.

FIGS. 12A and 12B show taps used for calculation of time activity.

FIG. 13 shows a tap used for calculation of space activity.

FIG. 14 illustrates threshold values for motion discrimination.

FIG. 15 is a flowchart for explaining the processing for setting amotion class code MCC of the motion discriminating circuit.

FIG. 16 illustrates pixels used for discrimination by majority decisionof the motion class code MCC.

FIG. 17 is a flowchart for explaining the processing for setting themotion class code MCC of the motion detecting circuit.

FIG. 18 shows an exemplary construction of a tap at an edge of an image.

FIG. 19 shows an exemplary construction of a tap at an edge of an image.

FIG. 20 illustrates pixels used for interpolation processing.

FIG. 21 Illustrates a pixel with its pixel value substituted.

FIG. 22 is a block diagram showing another structure of the lackingpixel creating circuit.

FIG. 23 shows the structure of an embodiment of an image processingdevice which generates a coefficient set used by the image processingdevice for selectively carrying out one or a plurality of modes, of animage processing mode for carrying out lacking pixel creation, an imageprocessing in consideration of chromatic aberration, and an imageprocessing mode in consideration of the telop position.

FIG. 24 illustrates chromatic aberration.

FIGS. 25A and 25B illustrate chromatic aberration.

FIGS. 26A to 26C illustrate switching of a tap.

FIG. 27 shows the structure of an embodiment of an image processingdevice which selectively carries out one or a plurality of modes, of animage processing mode for carrying out lacking pixel creation, an imageprocessing mode in consideration of chromatic aberration, and an imageprocessing mode in consideration of the telop position.

FIG. 28 is a flowchart for explaining the tap switching processingcorresponding to chromatic aberration.

FIGS. 29A to 29D show exemplary screens in which a telop or the like isdisplayed.

FIG. 30 is a flowchart for explaining the tap switching processingcorresponding to the telop position.

FIG. 31 illustrates a recording medium.

BEST MODE FOR CARRYING OUT THE INVENTION

Preferred embodiments of the present invention will now be described indetail with reference to the drawings.

FIG. 2 shows the structure of an embodiment of an image processingdevice according to the present invention. An effective pixel areacalculating circuit 11 generates an effective pixel area vertical flagVF and an effective pixel area horizontal flag HF indicating whetherpixels of an image inputted to a lacking pixel creating circuit 12 aresituated within an effective pixel area or not, on the basis of avertical synchronizing signal and a horizontal synchronizing signalsynchronized with the image inputted to the lacking pixel creatingcircuit 12, and outputs the effective pixel area vertical flag VF andthe effective pixel area horizontal flag HF to the lacking pixelcreating circuit 12. In the following description, the pixels are alsoreferred to as taps, and the pixel values are also referred to as tapdata.

The lacking pixel creating circuit 12 creates a pixel corresponding to alacking pixel included in the inputted image on the basis of a lackingflag LF corresponding to the inputted image and each pixel of the image,and the effective pixel area vertical flag VF and the effective pixelarea horizontal flag HF supplied from the effective pixel areacalculating circuit 11. The lacking pixel creating circuit 12substitutes the created pixel for the lacking pixel and thus outputs thecreated pixel.

FIG. 3 shows an exemplary structure of the effective pixel areacalculating circuit 11. A vertical synchronizing signal detectingcircuit 41 generates data indicating whether or not each pixel of theimage is within the effective pixel area in the vertical direction ofthe screen (hereinafter referred to as vertical effective pixel areadata) on the basis of the inputted vertical synchronizing signal, andsupplies the data to an effective area calculating circuit 43. Ahorizontal synchronizing signal detecting circuit 42 generates dataindicating whether or not each pixel of the image is within theeffective pixel area in the horizontal direction of the screen(hereinafter referred to as horizontal effective pixel area data) on thebasis of the inputted horizontal synchronizing signal, and supplies thedata to the effective area calculating circuit 43.

The effective area calculating circuit 43 corrects the verticaleffective pixel area data supplied from the vertical synchronizingsignal detecting circuit 41 and outputs the corrected data as aneffective pixel area vertical flag VF to the lacking pixel creatingcircuit 12.

For the effective pixel area vertical flag VF, a value 0 is set withinthe effective range of the display and a value 1 is set outside theeffective range of the display, for example, as shown in FIG. 4.

The effective area calculating circuit 43 corrects the horizontaleffective pixel area data supplied from the horizontal synchronizingsignal detecting circuit 42 and outputs the corrected data as aneffective pixel area horizontal flag HF to the lacking pixel creatingcircuit 12.

For the effective pixel area horizontal flag HF, a value 0 is set withinthe effective range of the display and a value 1 is set outside theeffective range of the display, for example, as shown in FIG. 4.

The lacking pixel creating circuit 12 can learn whether each pixel ofthe inputted image is situated within the effective pixel area or not,on the basis of the effective pixel area vertical flag VF and theeffective pixel area horizontal flag HF supplied from the effectivepixel area calculating circuit 11.

The lacking pixel creating circuit 12 will now be described. When theimaged inputted to the lacking pixel creating circuit 12 is aninterlaced image, the position of a pixel of a target field isvertically shifted by ½ from the position of a pixel of the fieldimmediately before or immediately after the target field.

The lacking pixel creating circuit 12 creates a pixel value of thelacking pixel on the basis of the pixel values of surrounding pixels inthe same field (field k in FIG. 5) as the target pixel of creation, thepixel values of pixels existing in the field immediately before (fieldk−1 in FIG. 5), and the pixel values of pixels existing in the fieldthat is two fields before (field k−2 in FIG. 5), as shown in FIG. 5, byclassification adaptive processing.

When the lacking pixel as a target of creation is situated at an edge ofthe image, lacking pixel creating circuit 12 selects only the pixelssituated within the effective range of the image (that is, discards thepixels situated outside the effective range of the image) on the basisof the effective pixel area vertical flag VF and the effective pixelarea horizontal flag HF supplied from the effective pixel areacalculating circuit 11, as shown in FIG. 6, and thus creates the pixelvalue of the lacking pixel on the basis of the selected pixels.

When the lacking pixel as a target of creation is situated at an edge ofthe image, the lacking pixel creating circuit 12 may also selecteffective pixels as taps by adaptively switching to a tap structure inwhich the pixels situated within the effective range of the image areadopted, on the basis of the effective pixel area vertical flag VF andthe effective pixel area horizontal flag HF supplied from the effectivepixel area calculating circuit 11, as shown in FIG. 7, and thus maycreate the pixel value of the lacking pixel.

FIG. 8 is a block diagram showing an exemplary structure of the lackingpixel creating circuit 12. The pixel value and the lacking flag LFindicating the lack of the pixel, inputted to the lacking pixel creatingcircuit 12, are supplied to a preprocessing circuit 101 and a tapconstructing circuit 102-1.

The effective pixel area vertical flag VF and the effective pixel areahorizontal flag HF inputted from the effective pixel area calculatingcircuit 11 are supplied to the preprocessing circuit 101, the tapconstructing circuits 102-1 to 102-5, a class combining circuit 107, anda coefficient-holding class code selecting circuit 109.

The preprocessing circuit 101 sets the lacking flag LF of the pixelsituated outside the effective pixel area on the basis of the effectivepixel area vertical flag VF and the effective pixel area horizontal flagHF. For example, a lacking flag LF of “1” indicates that the pixel valueis lacking, and a lacking flag LF of “0” indicates that the pixel valueis not lacking. The preprocessing Circuit 101 generates the value of thelacking pixel within the effective pixel area on the basis of thelacking flag LF corresponding to the pixel value and the pixel, by usinga linear interpolation filter, and sets the generated value for thelacking pixel. The preprocessing circuit 101 then supplies the set valueto the tap constructing circuits 102-1 to 102-5. That is, when a pixelor pixels arc lacking, the preprocessing circuit 101 increase the numberof prediction taps by the number of lacking pixels.

However, the class tap includes no lacking pixels and the classificationprocessing uses no pixel values that are generated by the preprocessingcircuit 101, as will be described later.

The processing at the preprocessing circuit 101 will now be describedwith reference to the flowchart of FIG. 9. At step S11, thepreprocessing circuit 101 discriminates whether a target pixel islacking or not on the basis of the lacking flag LF. If it is determinedthat the target pixel is not lacking, the processing goes to step S12,where the pixel value of the target pixel is set for the target pixeland then the processing ends.

If it is determined at step S11 that the target pixel is lacking, theprocessing goes to step S13 and the preprocessing circuit 101discriminates whether one of the two pixels adjacent to the target pixelin the horizontal direction is lacking or not on the basis of thelacking flag LF. If it is determined at step S13 that neither of the twopixels adjacent to the target pixel in the horizontal direction islacking, the processing goes to step S14 and the preprocessing circuit101 sets the average value of the pixel values of the two pixelsadjacent to the target pixel in the horizontal direction, as the pixelvalue of the target pixel. Then, the processing ends.

If it is determined at step S13 that one of the two pixels adjacent tothe target pixel in the horizontal direction is lacking, the processinggoes to step S15 and the preprocessing circuit 101 discriminates whetherboth of the two pixels adjacent to the target pixel in the horizontaldirection are lacking or not. If it is determined at step S15 that oneof the two pixels adjacent to the target pixel in the horizontaldirection is not lacking, the processing goes to step S16 and thepreprocessing circuit 101 sets the pixel value of the pixel that ishorizontally adjacent to the target pixel and that is not lacking, asthe pixel value of the target pixel. Then, the processing ends.

If it is determined at step S15 that both of the two pixels horizontallyadjacent to the target pixel are lacking, the processing goes to stepS17 and the preprocessing circuit 101 discriminates whether one of thetwo pixels adjacent to the target pixel in the vertical direction islacking or not on the basis of the lacking flag LF. If it is determinedat step ST17 that neither one of the two pixels adjacent to the targetpixel in the vertical direction is lacking, the processing goes to stepS18 and the preprocessing circuit 101 sets the average value of thepixel values of the two pixels vertically adjacent to the target pixel,as the pixel value of the target pixel. Then, the processing ends.

If it is determined at step S17 that one of the two pixels adjacent tothe target pixel in the vertical direction is lacking, the processinggoes to step S19 and the preprocessing circuit 101 discriminates whetherall the pixels adjacent to the target pixel are lacking or not on thebasis of the lacking flag LF. If it is determined at step S19 that oneof the pixels adjacent to the target pixel is not lacking, theprocessing goes to step S20 and the preprocessing circuit 101 sets thepixel value of the pixel that is adjacent to the target pixel and thatis not lacking, as the pixel value of the target pixel. Then, theprocessing ends.

If it is determined at step S19 that all the pixels adjacent to thetarget pixel are lacking, the processing goes to step S21 and thepreprocessing circuit 101 sets the pixel value of the pixel in the pastframe which is of the same position as the target pixel, as the pixelvalue of the target pixel. Then, the processing ends.

As described above, the preprocessing circuit 101 linearly interpolatesthe pixel value of the target pixel in the processing within theeffective pixel area, from the pixel values of the surrounding pixels.The interpolation processing by the preprocessing circuit 101 enablesexpansion of the range of taps that can be used in the subsequentprocessing.

The tap constructing circuit 102-1 sets the lacking flag LF of the pixelsituated outside the effective pixel area on the basis of the effectivepixel area vertical flag VF and the effective pixel area horizontal flagHF, then resets the lacking flag LF of the pixel situated outside theeffective pixel area, and supplies the lacking flag LF as a lacking flagtap SLFT1 to a motion class generating circuit 103. The tap constructingcircuit 102-1 selects a motion class tap TD1 consisting of the pixelswithin the effective pixel area that are not lacking, and supplies theselected motion class tap TD1 to the motion class generating circuit103.

The motion class generating circuit 103 generates a motion class codeMCC and a static/motion flag SMF on the basis of a parameter suppliedfrom an initializing circuit 111 and the lacking flag tap SLFT1 and theselected motion class tap TD1 supplied from the tap constructing circuit102-1, and outputs the motion class code MCC and the static/motion flagSMF to the tap constructing circuits 102-2 to 102-5 and a classcombining circuit 107. The motion class code MCC is 2-bit informationindicating the quantity of motion, and the static/motion flag SMF is1-bit information indicating the presence/absence of motion.

FIG. 10 shows the structure of the motion class generating circuit 103.The lacking flag tap SLFT1 and the motion class tap. TD1 supplied fromthe tap constructing circuit 102-1 are supplied to a motion detectingcircuit 151. The motion detecting circuit 151 generates a motion classcode MCC on the basis of the lacking flag tap SLFT1 and the motion classtap TD1 and outputs the motion class code MCC. The motion detectingcircuit 151 also supplies the generated motion class code MCC to astatic/motion discriminating circuit 152.

The structure of the motion detecting circuit 151 will now be describedwith reference to the block diagram of FIG. 11. A time activitycalculating circuit 181 calculates the time activity by adding absolutevalues of differences in the pixel values, for example, between 3×3pixels (included in the motion class tap TD1) that are within theeffective area, are not lacking and arc around the target pixel ofcreation, and corresponding 3×3 pixels (included in the motion class tapTD1) that are within the effective area and are not lacking, of theframe immediately before, on the basis of the lacking flag tap SLFT1 andthe motion class tap TD1 supplied from the tap constructing circuit102-1. The time activity calculating circuit 181 then supplies the timeactivity to a motion discriminating circuit 184. The time activitycalculating circuit 181 calculates the time activity by using only thepixels that are not lacking, without using the lacking pixels.

FIG. 12A shows an example of 3×3 pixels around the target pixel ofcreation, used for calculating the time activity. In FIG. 12A, “ERROR”indicates a lacking pixel. FIG. 12B shows an example of 3×3 pixels ofthe frame immediately before, corresponding to the pixels shown in FIG.12A. L1 to L3 in FIGS. 12A and 12B represent the respective lines, andthe same line number indicates the same position in the verticaldirection. H1 to H3 in FIGS. 12A and 12B represent the horizontalpositions of the respective pixels, and the same number indicates thesame position in the horizontal direction.

Since the lacking pixels are not used, the time activity in the case ofFIGS. 12A and 12B is calculated by the following equation (1).Time activity={|(q 2)−(p 2)|+|(q 3)−(p 3)|+|(q 4)−(p 4) +|(q 6)−(p6)|+|(q 7)−(p 7)|+|(q 9)−(p 9)|}/v  (1)In the equation (1), ( ) represents the pixel value of a pixel, ∥represents the function for finding an absolute value, and v representsthe number obtained by subtracting the number of lacking pixels from thenumber of pixels of a frame in which the target pixel of creationexists.

A space activity calculating circuit 182 calculates the space activity,for example, by adding 1 to the difference between the maximum value andthe minimum value of 3×3 pixels around the target pixel of creation, onthe basis of the lacking flag tap SLFT1 and the motion class tap TD1supplied from the tap constructing circuit 102-1, and supplies the spaceactivity to a threshold setting circuit 183.

FIG. 13 shows an example of 3×3 pixels around the lacking pixel as atarget of creation, used for calculating the space activity. The spaceactivity is calculated by the following equation (2).Space activity=Max(qi)−Min(qi)+1  (2)In the equation (2), Max(qi) represents the maximum value of the pixelvalues q1 to q9, and Min(qi) represents the minimum value of the pixelvalues q1 to q9.

The threshold setting circuit 183 selects a threshold value for motiondiscrimination, stored in advance in the threshold setting circuit 183,on the basis of the space activity supplied from the space activitycalculating circuit 182, and supplies the selected threshold value tothe motion discriminating circuit 184. As the threshold value for motiondiscrimination supplied to the motion discriminating circuit 184,various threshold values are selected depending on the value of thespace activity.

The motion discriminating circuit 184 sets a motion class code MCC onthe basis of the threshold value for motion discrimination supplied fromthe threshold setting circuit 183 and the time activity supplied fromthe time activity calculating circuit 181, and supplies the motion classcode MCC to a majority decision discriminating circuit 185, a delaycircuit 186 and a selector 187.

FIG. 14 shows threshold values for motion discrimination. As thethreshold value for motion discrimination, various threshold values areused depending on the space activity. When the space activity becomeslarge, a large threshold value is used. This is because when a pixelwith large space activity makes little motion, the time activity has alarge value.

The processing for setting the motion class code MCC by the motiondiscriminating circuit 184 will now be described with reference to theflowchart of FIG. 15. At step S31, the motion discriminating circuit 184discriminates whether or not the time activity is equal to or less thana threshold value 1. If it is determined that the time activity is equalto or less than the threshold value 1, the processing goes to step S32and the motion discriminating circuit 184 sets the motion class code MCCat 0. Then, the processing ends.

If it is determined at step S31 that the time activity exceeds thethreshold value 1, the processing goes to step S33 and the motiondiscriminating circuit 184 discriminates whether or not the timeactivity is equal to or less than a threshold value 2. If it isdetermined that the time activity is equal to or less than the thresholdvalue 2, the processing goes to step S34 and the motion discriminatingcircuit 184 sets the motion class code MCC at 1. Then, the processingends.

If it is determined at step S33 that the time activity exceeds thethreshold value 2, the processing goes to step S35 and the motiondiscriminating circuit 184 discriminates whether or not the timeactivity is equal to or less than a threshold value 3. If it isdetermined that the time activity is equal to or less than the thresholdvalue 3, the processing goes to step S36 and the motion discriminatingcircuit 184 sets the motion class code MCC at 2. Then, the processingends.

If it is determined at step S35 that the time activity exceeds thethreshold value 3, the processing goes to step S37 and the motiondiscriminating circuit 184 sets the motion class code MCC at 3. Then,the processing ends.

As described above, the motion discriminating circuit 184 sets themotion class code MCC on the basis of the threshold value and the timeactivity.

The majority decision discriminating circuit 185 sets the ultimatemotion class code MCC on the basis of the motion class codes MCC of aplurality of pixels. For example, as shown in FIG. 16, the majoritydecision discriminating circuit 185 sets the motion class code MCC ofthe target pixel on the basis of the motion class codes MCC of 14 pixelsaround the target pixel of creation.

The processing for setting the motion class code MCC by the motiondetecting circuit 151 will now be described with reference to theflowchart of FIG. 17. At step S51, the motion detecting circuit 151discriminates whether discrimination by majority decision will becarried out or not, in accordance with the setting of the parameter fromthe initializing circuit 111. If it is determined that discrimination bymajority decision will not be carried out, the processing goes to stepS52. The selector 187 selects the motion class code MCC of the targetpixel outputted from the motion discriminating circuit 184 and sets themotion class code MCC of the target pixel as the ultimate motion classcode MCC. Then, the processing ends.

If it is determined at step S51 that discrimination by majority decisionwill be carried out, the processing goes to step S53 and the majoritydecision discriminating circuit 185 discriminates whether or not thenumber of pixels for which a motion class code MCC of 3 is set, of the14 pixels, is larger than the threshold value 3. If it is determinedthat the number of pixels for which the motion class code MCC of 3 isset is larger than the threshold value 3, the processing goes to stepS54 and the majority decision discriminating circuit 185 sets the motionclass code MCC at 3. The selector 187 outputs the output of the majoritydecision discriminating circuit 185 as the ultimate motion class codeMCC, and then the processing ends.

If it is determined at step S53 that the number of pixels for which themotion class code MCC of 3 is set is equal to or less than the thresholdvalue 3, the processing goes to step S55 and the majority decisiondiscriminating circuit 185 discriminates whether or not the value of thesum of the number of pixels for which the motion class code MCC of 3 isset and the number of pixels for which the motion class code MCC of 2 isset, of the 14 pixels, is larger than the threshold value 2. If it isdetermined that the value of the sum of the number of pixels for whichthe motion class code MCC of 3 is set and the number of pixels for whichthe motion class code MCC of 2 is set is larger than the threshold value2, the processing goes to step S56 and the majority decisiondiscriminating circuit 185 sets the motion class code MCC at 2. Theselector 187 outputs the output of the majority decision discriminatingcircuit 185 as the ultimate motion class code MCC, and then theprocessing ends.

If it is determined at step S55 that the value of the sum of the numberof pixels for which the motion class code MCC of 3 is set and the numberof pixels for which the motion class code MCC of 2 is set is equal to orless than the threshold value 2, the processing goes to step S57 and themajority decision discriminating circuit 185 discriminates whether ornot the value of the sum of the number of pixels for which the motionclass code MCC of 3 is set, the number of pixels for which the motionclass code MCC of 2 is set and the number of pixels for which the motionclass code MCC of 1 is set, of the 14 pixels, is larger than thethreshold value 1. If it is determined that the value of the sum of thenumber of pixels for which the motion class code MCC of 3 is set, thenumber of pixels for which the motion class code MCC of 2 is set and thenumber of pixels for which the motion class code MCC of 1 is set islarger than the threshold value 1, the processing goes to step S58 andthe majority decision discriminating circuit 185 sets the motion classcode MCC at 1. The selector 187 outputs the output of the majoritydecision discriminating circuit 185 as the ultimate motion class codeMCC, and then the processing ends.

If it is determined at step S57 that the value of the sum of the numberof pixels for which the motion class code MCC of 3 is set, the number ofpixels for which the motion class code MCC of 2 is set and the number ofpixels for which the motion class code MCC of 1 is set is equal to orless than the threshold value 1, the processing goes to step S59 and themajority decision discriminating circuit 185 sets the motion class codeMCC at 0. The selector 187 outputs the output of the majority decisiondiscriminating circuit 185 as the ultimate motion class code MCC, andthen the processing ends.

In this manner, the motion detecting circuit 151 sets the ultimatemotion class code MCC on the basis of the motion class codes MCC of aplurality of pixels and the threshold values stored in advance.

As described above, the motion class generating circuit 103 sets themotion class code MCC from the pixel values of a plurality of pixels andoutputs the motion class code MCC to the static/motion discriminatingcircuit 152 and the lacking pixel creating circuit 12.

Referring again to FIG. 10, the static/motion discriminating circuit 152sets the static/motion flag SMF on the basis of the motion class codeMCC and outputs the static/motion flag SMF. For example, when the motionclass code MCC is 0 or 1, the static/motion flag SMF is set at 0. Whenthe motion class code MCC is 2 or 3, the static/motion flag SMF is setat 1.

The tap constructing circuit 102-2 selects an all-class prediction tapVET (not including the pixels outside the effective pixel area) coveringall the class structures, on the basis of the motion class code MCC andthe static/motion flag SMF supplied from the motion class generatingcircuit 103 and the lacking flag LF. The tap constructing circuit 102-2then supplies the all-class prediction tap VET to a variable tapselecting circuit 108.

The tap constructing circuit 102-3 sets the lacking flag LF of thepixels situated outside the effective pixel area on the basis of theeffective pixel area vertical flag VF and the effective pixel areahorizontal flag HF, then resets the lacking flag LF of the pixelssituated outside the effective pixel area, and supplies the lacking flagLF as a lacking flag tap SLFT2 to a DR class generating circuit 104. Thetap constructing circuit 102-3 selects a DR class tap TD2 that is withinthe effective pixel area and is not lacking, on the basis of the motionclass code MCC and the static/motion flag SMF supplied from the motionclass generating circuit 103 and the lacking flag LF, and supplies theselected DR class tap TD2 to the DR class generating circuit 104. The DRclass generating circuit 104 generates a DR class code DRCC decided inaccordance with a dynamic range, that is, the difference between themaximum pixel value and the minimum pixel value of the pixels that areincluded in the DR class tap TD2 and are not lacking, on the basis ofthe lacking flag tap SLFT2 and the DR class tap TD2 supplied from thetap constructing circuit 102-3. The DR class generating circuit 104 thenoutputs the DR class code DRCC to the class combining circuit 107. Thetap constructing circuit 102-4 sets the lacking flag LF of the pixelssituated outside the effective pixel area on the basis of the effectivepixel area vertical flag VF and the effective pixel area horizontal flagHF supplied from the effective pixel area calculating circuit 11, thenresets the lacking flag LF of the pixels situated outside the effectivepixel area, and supplies the lacking flag LF as a lacking flag tap SLFT3to a space class generating circuit 105. The tap constructing circuit102-4 selects a space class tap TD3 that is within the effective pixelarea and is not lacking, on the basis of the motion class code MCC andthe static/motion flag SMF supplied from the motion class generatingcircuit 103 and the lacking flag LF, and supplies the selected spaceclass lap TD3 to the space class generating circuit 105. The space classgenerating circuit 105 generates a space class code SCC corresponding tothe pixel value pattern on the basis of the lacking flag tap SLFT3 andthe space lap TD3 supplied from the tap constructing circuit 102-4, andoutputs the space class code SCC to the class combining circuit 107.

The tap constructing circuit 102-5 selects a lacking flag LF on thebasis of the effective pixel area vertical flag VF and the effectivepixel area horizontal flag HF supplied from the effective pixel areacalculating circuit 11, and supplies the selected lacking flag LF as alacking flag tap SLFT4 to a lacking class generating circuit 106. Thelacking class generating circuit 106 generates a lacking class code LCCon the basis of the lacking flag tap SLFT4 supplied from the tapconstructing circuit 102-5, and outputs the lacking class code LCC tothe class combining circuit 107.

The class combining circuit 107 combines the motion class code MCC, thestatic/motion flag SMF, the DR-class code DRCC, the space class code SCCand the lacking class code LCC on the basis of the effective pixel areavertical flag VF and the effective pixel area horizontal flag HFsupplied from the effective pixel area calculating circuit 11, thusforming a single ultimate class code CC. The class combining circuit 107then outputs the class code CC to the coefficient-holding class codeselecting circuit 109.

The coefficient-holding class code selecting circuit 109 generates aprediction tap selecting signal VT on the basis of the effective pixelarea vertical flag VF and the effective pixel area horizontal flag HFsupplied from the effective pixel area calculating circuit 11, acoefficient set and a prediction structure supplied from theinitializing circuit 111 and the class code CC supplied from the classcombining circuit 107, and supplies the generated prediction tapselecting signal VT to the variable tap selecting circuit 108. At thesame time, the coefficient-holding class code selecting circuit 109outputs a prediction coefficient W selected from the coefficient set onthe basis of the class code CC, to an estimate prediction operationcircuit 110. The coefficient set supplied from the initializing circuit111 is generated in advance, corresponding to the class found as aresult of classification by the class code CC, and is stored by theinitializing circuit 111.

The variable tap selecting circuit 108 selects a prediction tap ET onthe basis of the all-class prediction tap VET supplied from the tapconstructing circuit 102-2 and the prediction tap selecting circuit VTsupplied from the coefficient-holding class code selecting circuit 109,and supplies the selected prediction tap ET to the estimate predictionoperation circuit 110. For example, the variable tap selecting circuit108 selects a tap designated by the prediction tap selecting signal VT,from the taps included in the all-class prediction tap VET, anddesignates the selected tap as a prediction tap ET.

A product-sum operation unit 121 of the estimate prediction operationcircuit 110 calculates the pixel value of the lacking pixel using alinear estimate formula on the basis of the prediction tap ET suppliedfrom the variable tap selecting circuit 108 and the predictioncoefficient W supplied from the coefficient-holding class code selectingcircuit 109.

The product-sum operation unit 121 may also calculate the pixel value ofthe lacking pixel using a nonlinear estimate formula on the basis of theprediction coefficient W.

A filter 122 of the estimate prediction operation circuit 110 calculatesthe pixel value of the lacking pixel from the prediction tap ET suppliedfrom the variable tap selecting circuit 108.

The estimate prediction operation circuit 110 selects and outputs theoutput of the filter 122 or the output of the product-sum operation unit121 on the basis of the output mode set by the initializing circuit 111,thus generating the result corresponding to the output mode.

In this manner, the lacking pixel creating circuit 12 carries outclassification in accordance with the dynamic range, motion, lacking andpixel value pattern from the pixels within the effective pixel area, onthe basis of the effective pixel area vertical flag VF and the effectivepixel area horizontal flag HF, and calculates the lacking pixel value onthe basis of the pixel values of the pixels around the lacking pixel(not including the pixel values of the pixels outside the effectivepixel area).

The lacking pixel creating circuit 12 can improve the quality of aninputted image (for example, increase in gray scales (increase in thenumber of bits of Y data, U data and V data), elimination of noise,elimination of quantization distortion (including elimination ofdistortion in the time direction), creation of a resolution of quadrupledensity and so on), by switching the output mode of the estimateprediction operation circuit 110 to carry out the above-describedprocessing with respect to all the pixels.

Moreover, if it is determined that the lacking pixel as a target ofcreation is situated at an edge of the image and that a predeterminednumber of taps or more are lacking, the lacking pixel creating circuit12 may carry out linear interpolation processing on the basis of thepixel values of the adjacent pixels, thus interpolating the pixel valueof the lacking pixel, as shown in FIG. 20, instead of classificationadaptive processing.

When the lacking pixel as a target of creation is situated at an edge ofthe image and all the adjacent pixels are lacking, the lacking pixelcreating circuit 12 may set a value corresponding to a predetermineddull color (for example, gray) as the pixel value of the lacking pixel,or may set the pixel value of the pixel at the same position in the pastframe, as shown in FIG. 21.

FIG. 22 is a block diagram showing another structure of the lackingpixel creating circuit 12, which carries out the processing shown inFIG. 20 or 21. The pixel value and a lacking flag LF indicating the lackof the pixel as the data inputted to the lacking pixel creating circuit12 are supplied to a preprocessing circuit 201 and a tap constructingcircuit 202-1.

The preprocessing circuit 201 carries out the processing similar to thatof the preprocessing circuit 101. The preprocessing circuit 201generates a value of the lacking pixel by using a linear interpolationfilter on the basis of the inputted pixel value and the lacking flag LFindicating the lack of the pixel, then sets the generated value as thepixel value of the lacking pixel, and supplies the value to tapconstructing circuits 202-2 to 202-5.

The tap constructing circuit 202-1 supplies the lacking flag LF of theselected pixel as a lacking flag tap SLFT1 to a motion class generatingcircuit 203. The tap constructing circuit 202-1 selects a motion classtap TD1 consisting of pixels that area within the effective pixel rangeand are not lacking, and supplies the selected motion class tap TD1 tothe motion class generating circuit 203.

The motion class generating circuit 203 generates a motion class codeMCC and a static/motion flag SMF on the basis of a parameter suppliedfrom an initializing circuit 211 and the lacking flag LF and theselected motion class tap TD1 supplied from the tap constructing circuit202-1, and outputs the motion class code MCC and the static/motion flagSMF to the tap constructing circuits 202-2 to 202-5 and a classcombining circuit 207. The motion class code MCC is 2-bit informationindicating the quantity of motion, and the static/motion flag SMF is1-bit information indicating the presence/absence of motion. Forexample, when the motion class code MCC is 0 or 1, the static/motionflag SMF is set at 0. When the motion class code MCC is 2 or 3, thestatic/motion flag SMF is set at 1.

The tap constructing circuit 202-2 selects an all-class prediction tapVET (not including the pixels outside the effective pixel area) coveringall the class structures, on the basis of the motion class code MCC andthe static/motion flag SMF supplied from the motion class generatingcircuit 203 and the lacking flag LF. The tap constructing circuit 202-2then supplies the all-class prediction tap VET to a variable tapselecting circuit 208.

The tap constructing circuit 202-3 supplies the selected lacking flag LFas a lacking flag tap SLFT2 to a DR class generating circuit 204. Thetap constructing circuit 202-3 selects a DR class tap TD2 that is withinthe effective pixel area and is not lacking, on the basis of the motionclass code MCC and the static/motion flag SMF supplied from the motionclass generating circuit 203 and the lacking flag LF, and supplies theselected DR class tap TD2 to the DR class generating circuit 204. The DRclass generating circuit 204 generates a DR class code DRCC decided inaccordance with a dynamic range, that is, the difference between themaximum pixel value and the minimum pixel value of the pixels that arenot lacking, on the basis of the lacking flag tap SLFT2 and the DR classtap TD2 supplied from the tap constructing circuit 202-3. The DR classgenerating circuit 204 then outputs the DR class code DRCC to the classcombining circuit 207.

The tap constructing circuit 202-4 supplies the selected lacking flag LFas a, lacking flag tap SLFT3 to a space class generating circuit 205.The tap constructing circuit 202-4 selects a space class tap TD3 that iswithin the effective pixel area and is not lacking, on the basis of themotion class code MCC and the static/motion flag SMF supplied from themotion class generating circuit 203 and the lacking flag LF, andsupplies the selected space class tap TD3 to the space class generatingcircuit 205. The space class generating circuit 205 generates a spaceclass code SCC corresponding to the pixel value pattern on the basis ofthe lacking flag tap SLFT3 and the space tap TD3 supplied from the tapconstructing circuit 202-4, and outputs the space class code SCC to theclass combining circuit 207.

The tap constructing circuit 202-5 selects a lacking flag LF andsupplies the selected lacking flag LF as a lacking flag tap SLFT4 to alacking class generating circuit 206. The lacking class generatingcircuit 206 generates a Jacking class code LCC on the basis of thelacking flag tap SLFT4 supplied from the tap constructing circuit 202-5,and outputs the lacking class code LCC to the class combining circuit207.

The class combining circuit 207 combines the motion class code MCC, thestatic/motion flag SMF, the DR class code DRCC, the space class code SCCand the lacking class code LCC to form a single ultimate class code CC,and outputs the class code CC to a coefficient-holding class codeselecting circuit 209.

The coefficient-holding class code selecting circuit 209 generates aprediction tap selecting signal VT on the basis of a coefficient set anda prediction structure supplied from the initializing circuit 211 andthe class code CC supplied from the class combining circuit 207, andsupplies the generated prediction tap selecting signal VT to thevariable tap selecting circuit 208. At the same time, thecoefficient-holding class code selecting circuit 209 outputs aprediction coefficient W selected from the coefficient set on the basisof the class code CC, to an estimate prediction operation circuit 210.

The variable tap selecting circuit 208 selects a prediction tap ET onthe basis of the all-class prediction tap VET supplied from the tapconstructing circuit 202-2 and the prediction tap selecting circuit VTsupplied from the coefficient-holding class code selecting circuit 209,and supplies the selected prediction tap ET to the estimate predictionoperation circuit 210.

The estimate prediction operation circuit 210 calculates the pixel valueof the lacking pixel using a linear estimate formula on the basis of theprediction tap ET supplied from the variable tap selecting circuit 208and the prediction coefficient W supplied from the coefficient-holdingclass code selecting circuit 209, and outputs the calculated pixel valueto a selecting circuit 214.

The estimate prediction operation circuit 210 is equivalent to theproduct-sum operation unit 121 of FIG. 8.

A substituting circuit 212 sets a value corresponding to a predetermineddull color (for example, gray) as the pixel value of the lacking pixelon the basis of the lacking flag LF indicating the lack of the pixel,and supplies the value to the selecting circuit 214.

A linear interpolation circuit 213 carries out the processing similar tothat of the preprocessing circuit 210. The linear interpolation circuit213 generates a value of the lacking pixel by using a linearinterpolation filter on the basis of the inputted pixel value and thelacking flag LF indicating the lack of the pixel, then sets the value asthe pixel value of the lacking pixel, and supplies the value to theselecting circuit 214.

The substituting circuit 212 and the linear interpolation circuit 213are equivalent to the filter 122 of FIG. 8.

The selecting circuit 214 selects one of the outputs from the estimateprediction operation circuit 210, the substituting circuit 212 and thelinear interpolation circuit 213 on the basis of the effective pixelarea vertical flag VF and the effective pixel area horizontal flag HFsupplied from the effective pixel area calculating circuit 11, andoutputs the selected output as the output of the lacking pixel creatingcircuit 12.

By thus carrying out classification adaptive processing in accordancewith changes of the dynamic range, motion, lacking and pixel value, thelacking pixel creating circuit 12 can calculate the pixel value of thelacking pixel on the basis of the pixel values of the pixels around thelacking pixel, and also can interpolate or replace the lacking pixelsituated at an edge of the effective pixel area and then output theresultant pixel.

Depending on the position of the lacking pixel as a target of creation,relative to the edge of the image, the lacking pixel creating circuit 12may suitably switch to the processing described with reference to FIGS.6, 7, 18 to 21.

Although the class tap includes no lacking pixels in the abovedescription, the pixel value generated by the preprocessing circuit 101may be included in the class tap and the pixel value generated by thepreprocessing circuit 101 may be used for the classification processing.

As described above, the image processing device according to the presentinvention can constantly generate an image of higher quality regardlessof the positions of pixels on the screen. For example, the imageprocessing device can re-create a lacking pixel with little errorsregardless of the position of the lacking pixel on the screen.

An image processing device for generating in advance a coefficient setused for the image processing device which selectively carries out oneor a plurality of modes of an image processing mode for carrying outlacking pixel creation shown in the example of FIG. 2, an imageprocessing mode in consideration of chromatic aberration, and an imageprocessing mode in consideration of the telop position, will now bedescribed.

FIG. 23 shows the structure of an embodiment of the image processingdevice for generating a coefficient set in advance.

An image inputted to the image processing device is supplied to a downfilter 303 and a normal equation operation circuit 310.

A display position calculating circuit 301 calculates the distance ofeach pixel of the image from the center of the screen and suppliesposition information indicating the distance of each pixel of the imagefrom the center of the screen, to tap constructing circuits 304-1 to304-N.

The display position calculating circuit 301 may also supply theposition information indicating the distance of each pixel from centerof the screen, to a structure switching control circuit 302.

An initializing circuit 312 supplies image end information, aberrationinformation, processing mode and telop position information to thestructure switching control circuit 302.

When the processing mode indicates the creation of a lacking pixel, thestructure switching control circuit 302 supplies a tap selecting signalTS1, a tap selecting signal TS2 and tap selecting signals TS3-1 toTS3-(N-2) corresponding to the image end information to the tapconstructing circuits 304-1 to 304-N; respectively. When the processingmode indicates the aberration mode, the structure switching controlcircuit 302 supplies a tap selecting signal TS1, a tap selecting signalTS2 and tap selecting signals TS3-1 to TS3-(N-2) corresponding to theaberration information to the tap constructing circuits 304-1 to 304-N,respectively. When the processing mode indicates the telop mode, thestructure switching control circuit 302 supplies a tap selecting signalTS1, a tap selecting signal TS2 and tap selecting signals TS3-1 toTS3-(N-2) corresponding to the telop position information to the tapconstructing circuits 304-1 to 304-N, respectively.

The structure switching control circuit 302 may also select a pluralityof processing modes of the three processing modes.

An example of the aberration mode will now be described.

For example, the tap selecting signal TS1, the tap selecting signal TS2and the tap selecting signals TS3-1 to TS3-(N-2) are constituted by asignal corresponding to red, a signal corresponding to green and signalscorresponding to blue, respectively, that is, signals corresponding toRGB.

The structure switching control circuit 302 calculates the distance ofeach pixel from the center of the screen on the basis of the physicaladdress of each pixel on the screen supplied from the display positioncalculating circuit 301, and generates an aberration class code CCAconsisting of a class code corresponding to red, a class codecorresponding to green and a class code corresponding to blue on thebasis of the calculated distance from the center of screen and theaberration information inputted from the initializing circuit 312. Thestructure switching control circuit 302 supplies the generatedaberration class code CCA to a class combining circuit 307.

The structure switching control circuit 302 stores in advance therelation between the physical address of each pixel on the screen andthe distance of each pixel from the center of the screen, and finds thedistance of each pixel from the center of the screen on the basis of thestored relation and the physical address of each pixel on the screensupplied from the display position calculating circuit 301.

Alternatively, the structure switching control circuit 302 may generatesan aberration class code CCA consisting of a class code corresponding tored, a class code corresponding to green and a class code correspondingto blue on the basis of the aberration information inputted from theinitializing circuit 312 and the distance from the center of the screensupplied from the display position calculating circuit 301, and maysupply the generated aberration class code CCA to the class combiningcircuit 307.

The structure switching control circuit 302 generates the aberrationclass code CCA, for example, by quantizing the quantity of aberration.

Chromatic aberration will now be described.

For example, when a white light enters obliquely to the optical axis ofa lens, as shown in FIG. 24, an image of a blue light included in thewhite light is formed at a position closer to the optical axis than animage of a yellow light is. The image of the yellow light included inthe white light is formed at a position farther from the optical axisthan the image of the blue light is and closer to the optical axis thanan image of a red light is. The image of the red light included in thewhite light is formed at a position farther from the optical axis thanthe image of the yellow light is. Such deviation of image formingposition among the blue-light image, the yellow-light image and thered-light image is referred to as chromatic aberration. Large chromaticaberration means a long distance between the image forming positions ofthe blue-light image, the yellow-light image and the red-light image.

Since the magnitude of chromatic aberration increases correspondingly tothe distance between the position of the image and the center of thescreen, the pixels on the circumference of a circle around the center ofthe screen have chromatic aberration of the same magnitude, as shown inFIG. 25A.

This relation between the distance from the center of the screen and themagnitude of chromatic aberration can be shown in the graph of FIG. 25B.That is, chromatic aberration increases non-linearly to the distancefrom the center of the screen.

The down filter 303 adopts the processing corresponding to aberration orthe processing such as jitter addition or noise addition for theinputted image, and supplies an image having a pixel value correspondingto aberration or a jitter-added or noise-added image to the tapconstructing circuits 304-1 to 304-N.

The tap constructing circuit 304-1 switches the tap structure for eachof red, green and blue on the basis of the position information suppliedfrom the display position calculating circuit 301 and the tap selectingsignal TS1 supplied from the structure switching control circuit 302.The tap constructing circuit 304-1 then selects pixels included in theimage supplied from the down filter 303 as a motion class tap TD1corresponding to each of red, green and blue, and supplies the selectedmotion class tap TD1 to a motion class generating circuit 305. Themotion class tap TD1 outputted from the tap constructing circuit 304-1consists of a tap corresponding to red, a tap corresponding to green anda tap corresponding to blue.

FIGS. 26A to 26C illustrate the tap structures for red, green and blue,respectively, at the tap constructing circuit 304-1. For example, thetap corresponding green is constituted by a tap based on a target pixelas the center, as shown in FIG. 26A.

The structure switching control circuit 302 generates correction vectorsfor red and blue with reference to green, as shown in FIG. 26B, on thebasis of the aberration information inputted from the initializingcircuit 312, and supplies the tap selecting signal TS1 including thegenerated correction vectors to the tap constructing circuit 304-1.

The tap constructing circuit 304-1 selects, for example, a correctiontarget pixel for red with reference to the target pixel on the basis ofthe position information indicating the distance of each pixel from thecenter of the screen supplied from the display position calculatingcircuit 301 and the correction vector for red included in the tapselecting signal TS1, and constructs the tap corresponding to redcentering on the correction target pixel, as shown in FIG. 26C.Similarly, the tap constructing circuit 304-1 selects a correctiontarget pixel for blue with reference to the target pixel on the basis ofthe position information indicating the distance of each pixel from thecenter of the screen supplied from the display position calculatingcircuit 301 and the correction vector for blue included in the tapselecting signal TS1, and constructs the tap corresponding to bluecentering on the correction target pixel.

The tap constructing circuit 304-1 may also construct a tap selectingsignal TS1 including a correction vector for red, a correction vectorfor green and a correction vector for blue with reference to the targetpixel, then construct a tap corresponding to red centering on acorrection target pixel for red on the basis of the position informationindicating the distance of each pixel from the center of the screensupplied from the display position calculating circuit 301 and thecorrection vector for red included in the tap selecting signal TS1, thenconstruct a tap corresponding to green centering on a correction targetpixel for green on the basis of the position information indicating thedistance of each pixel from the center of the screen supplied from thedisplay position calculating circuit 301 and the correction vector forgreen included in the tap selecting signal TS1, and construct a tapcorresponding to blue centering on a correction target pixel for blue onthe basis of the position information indicating the distance of eachpixel from the center of the screen supplied from the display positioncalculating circuit 301 and the correction vector for blue included inthe tap selecting signal TS1.

The motion class generating circuit 305 generates a motion class codeMCC including a motion class code corresponding to red, a motion classcode corresponding to green and a motion class code corresponding toblue, and a static/motion flag SMF including a static/motion flagcorresponding to red, a static/motion flag corresponding to green and astatic/motion flag corresponding to blue, on the basis of the parametersupplied from the initializing circuit 312 and the motion class tap TD1supplied from the tap constructing circuit 304-1, and outputs the motionclass code MCC and the static/motion flag SMF to the tap constructingcircuits 304-2 to 304-N and the class combining circuit 307.

The tap constructing circuit 304-2 switches the tap structure for eachof red, green and blue on the basis of the motion class code MCC and thestatic/motion flag SMF for each of red, green and blue supplied from themotion class generating circuit 305, the position information indicatingthe distance of each pixel from the center of the screen supplied fromthe display position calculating circuit 301, and the tap selectingsignal TS2 supplied from the structure switching control circuit 302.The tap constructing circuit 304-2 selects an all-class prediction tapVET including a tap corresponding to red, a tap corresponding green anda tap corresponding to blue, and supplies the selected all-classprediction tap VET to a variable tap selecting circuit 308.

The tap constructing circuit 304-3 switches the tap structure for eachof red, green and blue on the basis of the motion class code MCC and thestatic/motion flag SMF for each of red, green and blue supplied from themotion class generating circuit 305, the position information indicatingthe distance of each pixel from the center of the screen supplied fromthe display position calculating circuit 301, and the tap selectingsignal TS3-1 for each of red, green and blue supplied from the structureswitching control circuit 302. The tap constructing circuit 304-3selects a class tap TD2-1 including a tap corresponding to red, a tapcorresponding green and a tap corresponding to blue, and supplies theselected class tap TD2-1 to a class generating circuit 306-1.

The class generating circuit 306-1 generates a class code CC1 includinga class code corresponding to red, a class code corresponding to greenand a class code corresponding to blue on the basis of the class tapTD2-1 supplied from the tap constructing circuit 304-3, and outputs thegenerated class code CC1 to the class combining circuit 307. The classcode CC1 can be, for example, a class code corresponding to thedifference between the maximum pixel value and the minimum pixel valueincluded in the class tap TD2-1.

The tap constructing circuits 304-4 to 304-N select class taps TD2-2 toTD2-(N-2) each including a tap corresponding to red, a tap correspondingto green and a tap corresponding to blue, on the basis of the motionclass code. MCC and the static/motion flag SMF supplied from the motionclass generating circuit 305, the position information indicating thedistance of each pixel from the center of the screen supplied from thedisplay position calculating circuit 301, and the tap selecting signalsTS3-2 to TS3-(N-2) supplied from the structure switching control circuit302. The tap constructing circuits 304-4 to 304-N supply the selectedclass taps TD2-2 to TD2-(N-2) to class generating circuits 306-2 to306-(N-2), respectively.

The class generating circuits 306-2 to 306-(N-2) generate one of classcodes CC2 to CC(N-2) including a class code corresponding to red, aclass code corresponding to green and a class code corresponding to blueon the basis of one of the class taps TD2-2 to TD2-(N-2) supplied fromthe tap constructing circuits 304-3 to 304-N, and output the generatedone of the class codes CC2 to CC(N-2) to the class combining circuit307. One of the class codes CC2 to CC(N-2) may be, for example, a classcode corresponding to the pixel value pattern.

The class combining circuit 307 combines the class code corresponding tored included in the aberration class code CCA and the class codecorresponding to red included in the class codes CC1 to CC(N-2) to forma class code corresponding to red included in a single ultimate classcode TCC, on the basis of the class code corresponding to red includedin the motion class code MCC and the static/motion flag corresponding tored included in the static/motion flag SMF.

The class combining circuit 307 combines the class code corresponding togreen included in the aberration class code CCA and the class codecorresponding to green included in the class codes CC1 to CC(N-2) toform a class code corresponding to green included in the single ultimateclass code TCC, on the basis of the class code corresponding to greenincluded in the motion class code MCC and the static/motion flagcorresponding to green included in the static/motion flag SMF.

The class combining circuit 307 combines the class code corresponding toblue included in the aberration class code CCA and the class codecorresponding to blue included in the class codes CC1 to CC(N-2) to forma class code corresponding to blue included in the single ultimate classcode TCC, on the basis of the class code corresponding to blue includedin the motion class code MCC and the static/motion flag corresponding toblue included in the static/motion flag. SMF.

The class combining circuit 307 outputs the class code TCC including theclass code corresponding to red, the class code corresponding to greenand the class code corresponding to blue, to a class code selectingcircuit 309.

The class code selecting circuit 309 generates a prediction tapselecting signal VT including the tap corresponding to red, the tapcorresponding to green and the tap corresponding to blue on the basis ofthe class code TCC supplied from the class combining circuit 307. Theclass code selecting circuit 309 supplies the generated prediction tapselecting signal VT to the variable tap selecting circuit 308 andoutputs the class code TCC to the normal equation operation circuit 310.

The variable tap selecting circuit 308 selects a prediction tap ETincluding the tap corresponding to red, the tap corresponding to greenand the tap corresponding to blue on the basis of the all-classprediction tap VET supplied from the tap constructing circuit 304-2 andthe prediction tap selecting signal VT supplied from the class codeselecting circuit 309, and supplies the selected prediction tap ET tothe normal equation operation circuit 310.

On receiving the prediction tap ET, which is the learning data suppliedfrom the variable tap selecting circuit 308, and the input image, whichis the teacher data supplied from the down filter 303, the normalequation operation circuit 310 uses these data to calculate a predictioncoefficient W for minimizing an error by a minimum square method. Theprediction coefficient W includes a prediction coefficient correspondingto red, a prediction coefficient corresponding green and a predictioncoefficient corresponding to blue.

The prediction coefficient W calculated by the normal equation operationcircuit 310 will be briefly described hereinafter.

For example, it is considered to find a prediction value E[y] of a pixelvalue y of an original image (equivalent to an inputted image(hereinafter suitably referred to as teacher data)), using a linearcombination model prescribed by linear combination of pixel values(hereinafter suitably referred to as learning data) x₁, x₂, . . . of animage which has noise added thereto or which has a pixel valuecorresponding to aberration by passing through the down filter 303 andpredetermined prediction coefficients w₁, w₂, . . . In this case, theprediction value E[y] can be expressed by the following equation (3).E[y]=w ₁ x ₁ +w ₂ x ₂+  (3)

Thus, if a matrix W consisting of a set of prediction coefficients w, amatrix X consisting of a set of learning data, and a matrix Y′consisting of a set of prediction values E[y] are defined as follows forgeneralization, $X = \begin{matrix}{x_{11},} & {x_{1\quad 2},} & \ldots & x_{1n} \\{x_{21},} & {x_{22},} & \ldots & x_{2n} \\\vdots & \vdots & \vdots & \vdots \\{x_{m1},} & {x_{m2},} & \ldots & x_{mn}\end{matrix}$ $W = {{\begin{matrix}W_{1} \\W_{2} \\\vdots \\W_{n}\end{matrix}\quad Y^{\prime}} = \begin{matrix}{E\left\lbrack y_{1} \right\rbrack} \\{E\left\lbrack y_{2} \right\rbrack} \\\vdots \\{E\left\lbrack y_{m} \right\rbrack}\end{matrix}}$the following observation equation is obtained.XW=Y′  (4)

Then, it is considered to find the prediction value E[y] close to thepixel value y of the original image by adopting the minimum squaremethod for this observation equation. In this case, if a matrix Yconsisting of a set of pixel values y of the original image and a matrixE consisting of a set of residuals e of the prediction value E[y] withrespect to the pixel value y of the original image are defined asfollows, $E = {{\begin{matrix}e_{1} \\c_{2} \\\ldots \\e_{m}\end{matrix}\quad Y} = \begin{matrix}y_{1} \\y_{2} \\\ldots \\y_{m}\end{matrix}}$the following residual equation is obtained from the equation (4).XW=Y+E  (5)

In this case, a prediction coefficient w_(i) for finding the predictionvalue E[y] close to the pixel value y of the original image can be foundby minimizing the following square error.$\sum\limits_{i = 1}^{m}e_{i}^{2}$

Therefore, if the above-described square error differentiated by theprediction coefficient w_(i) is 0, the prediction coefficient w_(i)satisfying the following equation is an optimum value for finding theprediction value E[y] close to the pixel value y of the original image.$\begin{matrix}{{{e_{1}\frac{\partial e_{1}}{\partial w_{i}}} + {e_{2}\frac{\partial e_{2}}{\partial w_{i}}} + \ldots + {e_{m}\frac{\partial e_{m}}{\partial w_{i}}}} = {0\left( {{i = 1},2,\ldots\quad,n} \right)}} & (6)\end{matrix}$

Thus, first, differentiating the equation (5) by the predictioncoefficient w_(i) provides the following equation. $\begin{matrix}{{\frac{\partial e_{i}}{\partial w_{1}} = {{x_{{i1},}\frac{\partial e_{i}}{\partial w_{2}}} = x_{i2}}},\ldots\quad,{\frac{\partial e_{i}}{\partial w_{n}} = {x_{in}\left( {{i = 1},2,{\ldots\quad n}} \right)}}} & (7)\end{matrix}$

From the equations (6) and (7), the following equation (8) is obtained.$\begin{matrix}{{{\sum\limits_{i = 1}^{m}{e_{i}x_{i1}}} = 0},{{\sum\limits_{i = 1}^{m}{e_{i}x_{i2}}} = 0},\ldots\quad,{{\sum\limits_{i = 1}^{m}{e_{i}x_{in}}} = 0}} & (8)\end{matrix}$

Moreover, in consideration of the relation of the learning data x, theprediction coefficient w, the teacher data y and the residual e in theresidual equation (5), the following normal equation can be obtainedfrom the equation (8). $\begin{matrix}{{{{\left( {\sum\limits_{i = 1}^{m}{x_{i1}x_{i1}}} \right)W_{1}} + {\left( {\sum\limits_{i = 1}^{m}{x_{i1}x_{i2}}} \right)W_{2}} + \ldots + \left( {\sum\limits_{i = 1}^{m}{x_{i1}x_{in}}} \right)} = \left( {\sum\limits_{i = 1}^{m}{x_{i1}y_{i}}} \right)}{{{\left( {\sum\limits_{i = 1}^{m}{x_{i2}x_{i1}}} \right)W_{1}} + {\left( {\sum\limits_{i = 1}^{m}{x_{i2}x_{i2}}} \right)W_{2}} + \ldots + \left( {\sum\limits_{i = 1}^{m}{x_{i2}x_{in}}} \right)} = \left( {\sum\limits_{i = 1}^{m}{x_{i2}y_{i}}} \right)}{{{\left( {\sum\limits_{i = 1}^{m}{x_{in}x_{i1}}} \right)W_{1}} + {\left( {\sum\limits_{i = 1}^{m}{x_{in}x_{i2}}} \right)W_{2}} + \ldots + \left( {\sum\limits_{i = 1}^{m}{x_{iN}x_{in}}} \right)} = \left( {\sum\limits_{i = 1}^{m}{x_{iN}y_{i}}} \right)}} & (9)\end{matrix}$

The optimum prediction coefficient w can be found by solving the normalequation (9). In solving the equation (9), it is possible to adopt, forexample, a sweep method (Gauss-Jordan elimination method).

Specifically, for example, it is now assumed that the pixel values ofthe prediction tap ET included in the learning data are x₁, x₂, x₃, . .. and that the prediction coefficients W to be found are w₁, w₂, w₃, . .. To find the pixel value y of a certain pixel of the teacher data bylinear combination of x₁, x₂, x₃, . . . and w₁, w₂, w₃, . . . , theprediction coefficients w₁, w₂, w₃, . . . must satisfy the followingequation.y=w ₁ x ₁ +w ₂x₂ +w ₃ x ₃+ . . .

Thus, the normal equation operation circuit 310 finds the predictioncoefficients w₁, w₂, w₃, . . . which minimize a square error of theprediction value w₁ x₁+w₂x₂+w₃ x₃+ . . . relative to the true value y,from the prediction tap of the same class and the pixels of thecorresponding teacher data by setting up and solving the above-describednormal equation (9).

More specifically, the normal equation operation circuit 310 finds theprediction coefficients w₁, w₂, w₃, . . . corresponding to red whichminimize a square error of the prediction value w₁x₁+w₂x₂+w₃x₃+ . . .relative to the true value y corresponding to red, from the predictiontap of the same class corresponding to red and the red component of thecorresponding teacher data by setting up and solving the normalequation.

The normal equation operation circuit 310 finds the predictioncoefficients w₁, w₂, w₃, . . . corresponding to green which minimize asquare error of the prediction value w₁x₁+w₂x₂+w₃x₃+ . . . relative tothe true value y corresponding to green, from the prediction tap of thesame class corresponding to green and the green component of thecorresponding teacher data by setting up and solving the normalequation.

The normal equation operation circuit 310 finds the predictioncoefficients w₁, w₂, w₃, . . . corresponding to blue which minimize asquare error of the prediction value w₁x₁+w₂x₂+w₃x₃+ . . . relative tothe true value y corresponding to blue, from the prediction tap of thesame class corresponding to blue and the blue component of thecorresponding teacher data by setting up and solving the normalequation.

Therefore, by carrying out this processing for each class, theprediction coefficients W including the prediction coefficientcorresponding to red, the prediction coefficient corresponding to greenand the prediction coefficient corresponding to blue are generated foreach class.

The prediction coefficients W including the prediction coefficientcorresponding to red, the prediction coefficient corresponding to greenand the prediction coefficient corresponding to blue for each class,found by the normal equation operation circuit 310, are suppliedtogether with the class code TCC to a coefficient memory 311. Thus, thecoefficient memory 311 stores the prediction coefficients W from thenormal equation operation circuit 310, at the address corresponding tothe class indicated by the class code TCC.

As described above, the image processing device shown in FIG. 23 cangenerate a coefficient set used for the image processing device whichselectively carries out one or a plurality of modes of the imageprocessing mode for carrying out lacking pixel creation, the imageprocessing mode in consideration of chromatic aberration and the imageprocessing mode in consideration of the telop position.

The image processing device shown in FIG. 23 may also generate acoefficient set used for the image processing device which carries outimage processing in the image processing mode in consideration ofchromatic aberration, where an image shot through a lens havingaberration and the same image shot through a lens having littleaberration are obtained, with the former image used as learning data andthe latter image used as teacher data.

FIG. 27 shows the structure of an embodiment of the image processingdevice according to the present invention, which selectively carries outone or a plurality of modes of the image processing mode for carryingout lacking pixel creation, the image processing mode in considerationof chromatic aberration and the image processing mode in considerationof the telop position, using the coefficient set generated by the imageprocessing device shown in FIG. 23.

A display position calculating circuit 401 calculates the distance ofeach pixel of an inputted image from the center of the screen andsupplies position information indicating the distance of each pixel fromthe center of the screen, to tap constructing circuits 403-1 to 403-N.

The display position calculating circuit 401 may also supply theposition information indicating the distance of each pixel of the imagefrom center of the screen, to a structure switching control circuit 402.

An initializing circuit 410 supplies image end information, aberrationinformation, processing mode and telop position information to thestructure switching control circuit 402.

When the processing mode indicates the creation of a lacking pixel, thestructure switching control circuit 402 supplies a tap selecting signalTS1, a tap selecting signal TS2 and tap selecting signals TS3-1 toTS3-(N-2) corresponding to the image end information to the tapconstructing circuits 403-1 to 403-N, respectively. When the processingmode indicates the aberration mode, the structure switching controlcircuit 402 supplies a tap selecting signal TS1, a tap selecting signalTS2 and tap selecting signals TS3-1 to TS3-(N-2) corresponding to theaberration information to the tap constructing circuits 403-1 to 403-N,respectively. When the processing mode indicates the telop mode, thestructure switching control circuit 402 supplies a tap selecting signalTS1, a tap selecting signal TS2 and tap selecting signals TS3-1 toTS3-(N-2) corresponding to the telop position information to the tapconstructing circuits 403-1 to 403-N, respectively.

The structure switching control circuit 402 may also select a pluralityof processing modes of the three processing modes.

An example of the aberration mode will now be described.

For example, the tap selecting signal TS1, the tap selecting signal TS2and the tap selecting signals TS3-1 to TS3-(N-2) are constituted by asignal corresponding to red, a signal corresponding to green and signalscorresponding to blue, respectively, that is, signals corresponding toRGB.

The structure switching control circuit 402 generates an aberrationclass code CCA consisting of a class code corresponding to red, a classcode corresponding to green and a class code corresponding to blue onthe basis of the aberration information inputted from the initializingcircuit 410, and supplies the generated aberration class code CCA to aclass combining circuit 406.

The tap constructing circuit 403-1 switches the tap structure for eachof red, green and blue on the basis of the position information suppliedfrom the display position calculating circuit 401 and the tap selectingsignal TS1 supplied from the structure switching control circuit 402.The tap constructing circuit 403-1 then selects pixels included in theimage supplied from a preprocessing circuit 403 as a motion class tapTD1 corresponding to each of red, green and blue, and supplies theselected motion class tap TD1 to a motion class generating circuit 404.The motion class tap TD1 outputted from the tap constructing circuit403-1 consists of a tap corresponding to red, a tap corresponding togreen and a tap corresponding to blue.

The motion class generating circuit 404 generates a motion class codeMCC including a motion class code corresponding to red, a motion classcode corresponding to green and a motion class code corresponding toblue, and a static/motion flag SMF including a static/motion flagcorresponding to red, a static/motion flag corresponding to green and astatic/motion flag corresponding to blue, on the basis of the parametersupplied from the initializing circuit 410 and the motion class tap TD1supplied from the tap constructing circuit 403-1, and outputs the motionclass code MCC and the static/motion flag SMF to the tap constructingcircuits 403-2 to 403-N and the class combining circuit 406.

The tap constructing circuit 403-2 switches the tap structure for eachof red, green and blue on the basis of the motion class code MCC and thestatic/motion flag SMF for each of red, green and blue supplied from themotion class generating circuit 404 and the tap selecting signal TS2supplied from the structure switching control circuit 402. The tapconstructing circuit 403-2 selects an all-class prediction tap VETincluding a tap corresponding to red, a tap corresponding green and atap corresponding to blue, and supplies the selected all-classprediction tap VET to a variable tap selecting circuit 407.

The tap constructing circuit 403-3 switches the tap structure for eachof red, green and blue on the basis of the motion class code MCC and thestatic/motion flag SMF for each of red, green and blue supplied from themotion class generating circuit 404 and the tap selecting signal TS3-1for each of red, green and blue supplied from the structure switchingcontrol circuit 402. The tap constructing circuit 403-3 selects a classtap TD2-1 including a tap corresponding to red, a tap correspondinggreen and a tap corresponding to blue, and supplies the selected classtap TD2-1 to a class generating circuit 405-1.

The class generating circuit 405-1 generates a class code CC1 includinga class code corresponding to red, a class code corresponding to greenand a class code corresponding to blue on the basis of the class tapTD2-1 supplied from the tap constructing circuit 403-3, and outputs thegenerated class code CC1 to the class combining circuit 406. The classcode CC1 can be, for example, a code corresponding to the differencebetween the maximum pixel value and the minimum pixel value of thepixels included in the class tap TD2-1.

The tap constructing circuits 403-4 to 403-N select one of class tapsTD2-2 to TD2-(N-2) each including a tap corresponding to red, a tapcorresponding to green and a tap corresponding to blue, on the basis ofthe motion class code MCC and the static/motion flag SMF supplied fromthe motion class generating circuit 404 and the tap selecting signalsTS3-2 to TS3-(N-2) supplied from the structure switching control circuit402, and supply the selected one of the class taps TD2-2 to TD2-(N-2) toclass generating circuits 405-2 to 405-(N-2).

The class generating circuits 405-2 to 405-(N-2) generate one of classcodes CC2 to CC(N-2) including a class code corresponding to red, aclass code corresponding to green and a class code corresponding to blueon the basis of one of the class taps TD2-2 to TD2-(N-2) supplied fromone of the tap constructing circuits 403-3 to 403-N, and output thegenerated one of the class codes CC2 to CC(N-2) to the class combiningcircuit 406. The class code CC2 may be, for example, a class codecorresponding to the pixel value pattern.

The class combining circuit 406 combines the class code corresponding tored included in the aberration class code CCA and the class codecorresponding to red included in the class codes CC1 to CC(N-2) to forma class code corresponding to red included in a single ultimate classcode TCC, on the basis of the class code corresponding to red includedin the motion class code MCC and the static/motion flag corresponding tored included in the static/motion flag SMF.

The class combining circuit 406 combines the class code corresponding togreen included in the aberration class code CCA and the class codecorresponding to green included in the class codes CC1 to CC(N-2) toform a class code corresponding to green included in the single ultimateclass code TCC, on the basis of the class code corresponding to greenincluded in the motion class code MCC and the static/motion flagcorresponding to green included in the static/motion flag SMF.

The class combining circuit 406 combines the class code corresponding toblue included in the aberration class code CCA and the class codecorresponding to blue included in the class codes CC1 to CC(N-2) to forma class code corresponding to blue included in the single ultimate classcode TCC, on the basis of the class code corresponding to blue includedin the motion class code MCC and the static/motion flag corresponding toblue included in the static/motion flag SMF.

The class combining circuit 406 outputs the class code TCC including theclass code corresponding to red, the class code corresponding to greenand the class code corresponding to blue, to a coefficient-holding classcode selecting circuit 408.

The coefficient-holding class code selecting circuit 408 stores inadvance a prediction tap selecting signal VT and a coefficient setcorresponding to the class code TCC, which are supplied from theinitializing circuit 410.

The coefficient-holding class code selecting circuit 408 generates theprediction tap selecting signal VT including the tap corresponding tored, the tap corresponding to green and the tap corresponding to blue onthe basis of the class code TCC supplied from the class combiningcircuit 406, and supplies the generated prediction tap selecting signalVT to the variable tap selecting circuit 407. At the same time, thecoefficient-holding class code selecting circuit 408 outputs predictioncoefficients W consisting of a prediction coefficient corresponding tothe class code for red included in the class code TCC, a predictioncoefficient corresponding to the class code for green included in theclass code TCC and a prediction coefficient corresponding to the classcode for blue included in the class code TCC, to an estimate predictionoperation circuit 409.

The variable tap selecting circuit 407 selects a prediction tap ETincluding the tap corresponding to red, the tap corresponding to greenand the tap corresponding to blue on the basis of the all-classprediction tap VET supplied from the tap constructing circuit 403-2 andthe prediction tap selecting signal VT supplied from thecoefficient-holding class code selecting circuit 408, and supplies theselected prediction tap ET to the estimate prediction operation circuit409.

A product-sum operation unit 421 of the estimate prediction operationcircuit 409 calculates a red component of the pixel value using a linearestimate formula on the basis of the tap corresponding to red includedin the prediction tap ET supplied from the variable tap selectingcircuit 407 and the prediction coefficient corresponding to red includedin the prediction coefficients W supplied from the coefficient-holdingclass code selecting circuit 408.

The product-sum operation unit 421 of the estimate prediction operationcircuit 409 calculates a green component of the pixel value using alinear estimate formula on the basis of the tap corresponding to greenincluded in the prediction tap ET supplied from the variable tapselecting circuit 407 and the prediction coefficient corresponding togreen included in the prediction coefficients W supplied from thecoefficient-holding class code selecting circuit 408.

The product-sum operation unit 421 of the estimate prediction operationcircuit 409 calculates a blue component of the pixel value using alinear estimate formula on the basis of the tap corresponding to blueincluded in the prediction tap ET supplied from the variable tapselecting circuit 407 and the prediction coefficient corresponding toblue included in the prediction coefficients W supplied from thecoefficient-holding class code selecting circuit 408.

The product-sum operation unit 421 of the estimate prediction operationcircuit 409 may also calculate the pixel value of the lacking pixelusing a nonlinear estimate formula on the basis of the predictioncoefficients W.

In this manner, the image processing device shown in FIG. 27 canselectively carry out one or a plurality of modes of the imageprocessing mode for carrying out lacking pixel creation, the imageprocessing mode in consideration of chromatic aberration and the imageprocessing mode in consideration of the telop position, and can providea sharper image than the conventional device.

The tap switching processing corresponding to chromatic aberration inthe aberration mode in the image processing device shown in FIG. 23 willnow be described with reference to the flowchart of FIG. 28. At stepS101, the structure switching control circuit 302 obtains aberrationinformation supplied from the initializing circuit 312.

At step S102, the structure switching control circuit 302 selects atarget pixel. At step S103, the display position calculating circuit 301finds the relative distance between the target pixel and the center ofthe screen. At step S104, the structure switching control circuit 302generates a correction vector for red, a correction vector for green anda correction vector for blue, and supplies a tap selecting signal TS1including the correction vectors to the tap constructing circuit 304-1,a tap selecting signal TS2 including the correction vectors to the tapconstructing circuit 304-2, and tap selecting signals TS3-1 to TS3-(N-2)including the correction vectors to the tap constructing circuits 304-3to 304-N, respectively.

At step S105, the tap constructing circuit 304-1 switches the tap on thebasis of the position information indicating the relative distancebetween the target pixel and the center of the screen, and thecorrection vector for red, the correction vector for green and thecorrection vector for blue, and selects a motion class tap TD1corresponding to each of red, green and blue. The tap constructingcircuit 304-2 switches the tap on the basis of the position informationindicating the relative distance between the target pixel and the centerof the screen, and the correction vector for red, the correction vectorfor green and the correction vector for blue, and selects an all-classprediction tap VET corresponding to each of red, green and blue. The tapconstructing circuits 304-3 to 304-N switch the tap on the basis of theposition information indicating the relative distance between the targetpixel and the center of the screen, and the correction vector for red,the correction vector for green and the correction vector for blue, andrespectively select DR class taps TD2-1 to TD2-(N-2) corresponding toeach of red, green and blue.

At step S106, the image processing device discriminates whether or notthe processing is completed for all the pixels. If it is determined thatthe processing is not completed for all the pixels, the image processingdevice returns to step S102 and repeats the tap switching processing.

As described above, in the aberration mode, the image processing deviceshown in FIG. 23 can switch the tap correspondingly to the screenposition in consideration of aberration.

In the aberration mode, the image processing device shown in FIG. 27switches the tap correspondingly to the screen position in accordancewith the processing similar to the processing described with referenceto the flowchart of FIG. 28. Therefore, the processing will not bedescribed further in detail.

Another processing carried out by the image processing device shown inFIG. 23 and the image processing device shown in FIG. 27 will now bedescribed.

In the image processing device shown in FIG. 23, when the telop mode isdesignated by the initializing circuit 312, the structure switchingcontrol circuit 302 obtains telop position information indicating atelop display area for displaying a telop. The telop positioninformation indicates the position and size of the telop display areasuch as upper 30 lines, lower 50 lines, or right 100 pixels.

The structure switching control circuit 302 may also obtain dataindicating the telop display area from the inputted image.

FIGS. 29A to 29D show examples of the screen in which a telop or thelike is displayed. In the example of FIG. 29A, an image andcorresponding characters are displayed in telop display areas in theupper and lower parts of the image display area. Since the image in thetelop display areas includes a large quantity of flat parts and edgeparts, its signal characteristic are different from those of a naturalimage or the like.

In the example of FIG. 29B, characters displayed in a telop display areain the lower part of the image display area are caused to run on theimage, from the right side to the left side of the screen.

In the example of FIG. 29C, characters displayed in a telop display areaon the right half part of the screen are caused to run on the image,from the upper side to the lower side of the screen.

In the example of FIG. 29D, an image generated by computer graphics isdisplayed in a frame image display area on the four sides surroundingthe image display area.

An exemplary operation of the image processing device of FIG. 23 in thetelop mode will now be described.

The display position calculating circuit 301 calculates the physicaladdress on the screen of each pixel of an inputted image and suppliesthe calculated physical address to the tap constructing circuits 304-1to 304-N.

The structure switching control circuit 302 generates a tap selectingsignal TS1, a tap selecting signal TS2 and tap selecting signals TS3-1to TS3-(N-2) on the basis of the telop position information, andsupplies the tap selecting signal TS1 to the tap constructing circuit304-1, the tap selecting signal TS2 to the tap constructing circuit304-2, and the tap selecting signals TS3-1 to TS3-(N-2) to the tapconstructing circuits 304-3 to 304-N, respectively.

On the basis of the physical address of each pixel on the screen and thetap selecting signal TS1, the tap constructing circuit 304-1 selects,for example, a tap using a broader range of pixels when the target pixelbelongs to the image display area, and a tap using a narrower range ofpixels when the target pixel belongs to the telop display area. Thus,the tap constructing circuit 304-1 selects a motion class tap TD1. Forexample; by selecting a tap using a broader range of pixels when anatural image is displayed in the image display area, the imageprocessing device can carry out image processing using an imagecomponent which gently changes over a large number of pixels.

On the other hand, in the telop display area in which characters aredisplayed, the pixel values of pixels corresponding to the charactersare substantially the same value and the pixel values of pixelscorresponding to the background are substantially the same value. Forexample, the pixel values of pixels corresponding to the charactersdisplayed in white and the pixel values of pixels corresponding to thebackground displayed in black are largely different.

That is, in the telop display area, the value of a tap across thecharacters and the background changes abruptly. The value of a tapconsisting of only the characters or only the background changes little.Therefore, by selecting a tap of a narrower range for the telop displayarea, the image processing device can carry out classification oradaptive processing corresponding appropriately to an image with anabruptly changing pixel value.

When characters are displayed to run in the horizontal direction of thescreen in the telop display area as shown in FIG. 29B, by selecting ahorizontally long tap with respect to the telop display area, the imageprocessing device can carry out more effective image processing whichrealizes a high noise elimination effect even with less classes. Whencharacters are displayed to run tin the vertical direction of the screenin the telop display area as shown in FIG. 29C, the image processingdevice can carry out more effective image processing by selecting avertically long tap with respect to the telop display area.

In this manner, the image processing device carries out optimal signalprocessing by using different tap structures and prediction coefficientsW for the telop display area and the image display area.

The tap switching processing corresponding to the telop position in theimage processing device shown in FIG. 23 in the telop mode will now bedescribed with reference to the flowchart of FIG. 30.

At step S201, the structure switching control circuit 302 obtains telopposition information supplied from the initializing circuit 312. Thestructure switching control circuit 302 generates a tap selecting signalTS1, a tap selecting signal TS2 and tap selecting signals TS3-1 toTS3-(N-2) corresponding to the position of the telop, and supplies thetap selecting signal TS1 to the tap constructing circuit 304-1, the tapselecting signal TS2 to the tap constructing circuit 304-2, and the tapselecting signals TS3-1 to TS3-(N-2) to the tap constructing circuits304-3 to 304-N, respectively.

At step S202, the tap constructing circuit 304-1 selects a target pixel.The tap constructing circuit 304-2 selects a target pixel. The tapconstructing circuits 304-3 to 304-N select targets pixels,respectively.

At step S203, the tap constructing circuit 304-1 discriminates whetherthe target pixel is a pixel within the telop or not, on the basis of thephysical address of each pixel on the screen and the tap selectingsignal TS1. If it is determined that the target pixel is a pixel withinthe telop, the processing goes to step S204 and the tap constructingcircuit 304-1 switches the tap and selects a motion class tap TD1corresponding to the telop. Then, the processing goes to step S206.

If it is determined at step S203 that the target pixel is not a pixelwithin the telop, the processing goes to step S205 and the tapconstructing circuit 304-1 switches the tap and selects a motion classtap TD1 corresponding to a natural image. Then, the processing goes tostep S206.

At steps S203 to S205, the tap constructing circuits 304-2 to 304-Ncarry out the processing similar to that of the tap constructing circuit304-1. Therefore, the processing will not be described further indetail.

At step S206, the tap constructing circuits 304-1 to 304-N discriminatewhether or not the processing is completed for all the pixels. If it isdetermined that the processing is not completed for all the pixels, theprocessing returns to step S202 to repeat the tap switching processing.

If it is determined at step S206 that the processing is completed forall the pixels, the processing ends.

In this manner, in the telop mode, the image processing device shown inFIG. 23 can switch the tap correspondingly to whether or not the targetpixel belongs to the telop display area.

In the telop mode, the image processing device shown in FIG. 27 switchesthe tap correspondingly to whether or not the target pixel belongs tothe telop display area, in accordance with the processing similar to theprocessing described with reference to the flowchart of FIG. 30.Therefore, the processing will not be described further in detail.

With respect to the image shown in FIG. 29D, the image processing deviceshown in FIG. 23 or FIG. 27 carries out processing by switching the tapof the frame image display area and the tap of the image display area.

The above-described series of processing can be executed by hardware orcan be executed by software. When the series of processing is executedby software, a program constituting the software is installed from arecording medium, for example, to a general-purpose personal computercapable of executing various functions, by installing a computerincorporated in the hardware or various programs.

FIG. 31 illustrates an exemplary recording medium and computer. A CPU(central processing unit) 501 actually executes various applicationprograms and OS (operating system). A ROM (read-only memory) 502, ingeneral, basically stores fixed data of the programs and operationparameters used by the CPU 501. A RAM (random access memory) 503 storesthe programs used for execution by the CPU 501 and the parameterssuitably changing in the execution. These units are interconnected by ahost bus 504 made up of a CPU bus or the like.

The host bus 504 is connected to an external bus 506 such as a PCI(peripheral component interconnect/interface) via a bridge 505.

A keyboard 508 is operated by the user when the user inputs variousinstructions to the CPU 501. A mouse 509 is operated by the user whenthe user designates or selects a point on the screen of a display 510.The display 510 is made up of a liquid crystal display or a CRT (cathoderay tube) and displays various types of information as texts and images.An HDD (hard disk drive) 511 drives a hard disk, thus recording orreproducing the programs and information executed by the CPU 501 to orfrom the hard disk.

A drive 512 reads out data or a program recorded on a magnetic disk 551,an optical disc 552, a magneto-optical disc 553 or a semiconductormemory 554 loaded thereon, and supplies the data or program to the RAM503 connected via an interface 507, the external bus 506, the bridge 505and the host bus 504.

The units of the keyboard 508 to the drive 512 are connected to theinterface 507, and the interface 507 is connected to the CPU 501 via theexternal bus 506, the bridge 505 and the host bus 504.

The recording media are constituted not only by the removable media suchas the magnetic disk 551 (including a floppy disk), the optical disc 552(including CD-ROM (compact disc-read only memory) and DVD (digitalversatile disc)), the magneto-optical disc 553 (including MD(mini-disc)) and the semiconductor memory 554 having the programsrecorded thereon, which are distributed to provide the user with theprograms for executing the processing corresponding to the block diagramseparately from the computer, as shown in FIG. 23, but also by the ROM502 and the HDD 511 having the programs recorded thereon, which areincorporated in the computer in advance and thus provided for the user.

The programs for executing the processing corresponding to the blockdiagram for the user may also be supplied to the computer via a wired orradio communication medium.

In this specification, the steps describing the programs stored in therecording media include the processing which is carried out in timeseries along the described order and also the processing which is notnecessarily carried out in time series but is carried out in parallel orindividually.

As described above, with the image processing device and method and therecording medium according to the present invention, positioninformation indicating the position within a frame, of a target pixel ofan input image signal consisting of a plurality of pixels, is detected,and the class of the target pixel is determined from a plurality ofclasses in accordance with the position information. A plurality ofpixels are selected from the input image signal as a prediction tap, andarithmetic processing based on conversion data obtained in advance bylearning for each of the classes and the prediction tap is carried out,thus outputting an output image signal of higher quality than the inputimage signal. Therefore, an image of higher quality can be constantlygenerated regardless of the position of the pixel on the screen.

Moreover, with the image processing device and method and the recordingmedium according to the present invention, position informationindicating the position within a frame, of a target pixel of an inputimage signal consisting of a plurality of pixels, is detected, and aplurality of pixels having their positional relations with the targetpixel varied in accordance with the position information are selectedfrom the input image signal as a class tap. The class of the targetpixel is selected from a plurality of classes in accordance with theclass tap, and a plurality of pixels are selected from the input imagesignal as a prediction tap. Arithmetic processing based on conversiondata obtained in advance by learning for each of the classes and theprediction tap is carried out, thus outputting an output image signal ofhigher quality than the input image signal. Therefore, an image ofhigher quality can be constantly generated regardless of the position ofthe pixel on the screen.

Furthermore, with the image processing device and method and therecording medium according to the present invention, positioninformation indicating the position within a frame, of a target pixel ofan input image signal consisting of a plurality of pixels, is detected,and a plurality of pixels are selected from the input image signal as aclass tap. The class of the target pixel is determined from a pluralityof classes in accordance with the class tap, and a plurality of pixelshaving their positional relations with the target pixel varied inaccordance with the position information are selected from the inputimage signal as a prediction tap. Arithmetic processing based onconversion data obtained in advance by learning for each of the classesand the prediction tap is carried out, thus outputting an output imagesignal of higher quality than the input image signal. Therefore, animage of higher quality can be constantly generated regardless of theposition of the pixel on the screen.

In addition, with the image processing device and method and therecording medium according to the present invention, a plurality ofpixels are selected from an input image signal as a provisional classtap, for each target pixel of the input image signal consisting of aplurality of pixels, and a plurality of pixels having their positionalrelations with the target pixel varied in accordance with the positionof the provisional class tap within a frame arc selected from the inputimage signal as a true class tap. The class of the target pixel isdetermined from a plurality of classes on the basis of the true classtap, and a plurality of pixels are selected from the input image signalas a prediction tap. Arithmetic processing based on conversion dataobtained in advance by learning for each of the classes and theprediction tap is carried out, thus outputting an output image signal ofhigher quality than the input image signal. Therefore, an image ofhigher quality can be constantly generated regardless of the position ofthe pixel on the screen.

Moreover, with the image processing device and method and the recordingmedium according to the present invention, a plurality of pixels areselected from an input image signal as a class tap, for each targetpixel of the input image signal consisting of a plurality of pixels, andthe class of the target pixel is determined from a plurality of classeson the basis of the class tap. A plurality of pixels for said eachtarget pixel are selected from the input image signal as a provisionalprediction tap, and a plurality of pixels having their positionalrelations with the target pixel varied in accordance with the positionof the provisional prediction tap within a frame are selected from theinput image signal as a true prediction tap. Arithmetic processing basedon conversion data obtained in advance by learning for each of theclasses and the true prediction tap is carried out, thus outputting anoutput image signal of higher quality than the input image signal.Therefore, an image of higher quality can be constantly generatedregardless of the position of the pixel on the screen.

1. An image processing device comprising: position detecting means fordetecting position information indicating the position within a frame,of a target pixel of an input image signal consisting of a plurality ofpixels; class determining means for determining the class of the targetpixel from a plurality of classes in accordance with the positioninformation; prediction tap selecting means for selecting a plurality ofpixels from the input image signal as a prediction tap; and operationmeans for carrying out arithmetic processing based on conversion dataobtained in advance by learning for each of the classes and theprediction tap, thus outputting an output image signal of higher qualitythan the input image signal.
 2. The image processing device as claimedin claim 1, wherein the class determining means determines the class inaccordance with which of an effective area and an invalid area in theframe is indicated by the position information.
 3. The image processingdevice as claimed in claim 1, wherein the class determining meansdetermines the class on the basis of the distance from the center of theframe indicated by the position information.
 4. The image processingdevice as claimed in claim 1, wherein the class determining meansdetermines the class on the basis of whether or not the positioninformation indicates the coincidence with the position where a telop isinserted in the frame.
 5. The image processing device as claimed inclaim 1, further comprising target pixel determining means fordetermining the target pixel in accordance with the motion of aprovisional target pixel of the input image signal.
 6. The imageprocessing device as claimed in claim 5, further comprising motiondetecting means for detecting the motion of the provisional targetpixel.
 7. The image processing device as claimed in claim 1, wherein theoperation means outputs an image signal including information about alacking pixel included in the input image signal, as the output imagesignal.
 8. The image processing device as claimed in claim 1, whereinthe operation means outputs an image signal having less noise than theinput image signal, as the output image signal.
 9. The image processingdevice as claimed in claim 1, wherein the operation means outputs animage signal having a higher resolution than the input image signal, asthe output image signal.
 10. The image processing device as claimed inclaim 1, wherein the operation means outputs an image signal having lessdistortion due to the aberration of a lens than the input image signal,as the output image signal.
 11. An image processing method comprising: aposition detecting step of detecting position information indicating theposition within a frame, of a target pixel of an input image signalconsisting of a plurality of pixels; a class determining step ofdetermining the class of the target pixel from a plurality of classes inaccordance with the position information; a prediction tap selectingstep of selecting a plurality of pixels from the input image signal as aprediction tap; and an operation step of carrying out arithmeticprocessing based on conversion data obtained in advance by learning foreach of the classes and the prediction tap, thus outputting an outputimage signal of higher quality than the input image signal.
 12. Arecording medium having recorded thereon a program for causing acomputer to execute image processing, the program comprising: a positiondetecting step of detecting position information indicating the positionwithin a frame, of a target pixel of an input image signal consisting ofa plurality of pixels; a class determining step of determining the classof the target pixel from a plurality of classes in accordance with theposition information; a prediction tap selecting step of selecting aplurality of pixels from the input image signal as a prediction tap; andan operation step of carrying out arithmetic processing based onconversion data obtained in advance by learning for each of the classesand the prediction tap, thus outputting an output image signal of higherquality than the input image signal.
 13. An image processing devicecomprising: position detecting means for detecting position informationindicating the position within a frame, of a target pixel of an inputimage signal consisting of a plurality of pixels; class tap selectingmeans for selecting a plurality of pixels having their positionalrelations with the target pixel varied in accordance with the positioninformation, from the input image signal, as a class tap; classdetermining means for determining the class of the target pixel from aplurality of classes in accordance with the class tap; prediction tapselecting means for selecting a plurality of pixels from the input imagesignal as a prediction tap; and operation means for carrying outarithmetic processing based on conversion data obtained in advance bylearning for each of the classes and the prediction tap, thus outputtingan output image signal of higher quality than the input image signal.14. The image processing device as claimed in claim 13, wherein theclass tap selecting means selects, as the class tap, a plurality ofpixels varied in accordance with which of an effective area and aninvalid area in the frame is indicated by the position information. 15.The image processing device as claimed in claim 13, wherein the classtap selecting means selects, as the class tap, a plurality of pixelsvaried on the basis of the distance from the center of the frameindicated by the position information.
 16. The image processing deviceas claimed in claim 13, wherein the class tap selecting means selects,as the class tap, a plurality of pixels varied on the basis of whetheror not the position information indicates the coincidence with theposition where a telop is inserted in the frame.
 17. The imageprocessing device as claimed in claim 13, further comprising targetpixel determining means for determining the target pixel in accordancewith the motion of a provisional target pixel of the input image signal.18. The image processing device as claimed in claim 17, furthercomprising motion detecting means for detecting the motion of theprovisional target pixel.
 19. The image processing device as claimed inclaim 13, wherein the operation means outputs an image signal includinginformation about a lacking pixel included in the input image signal, asthe output image signal.
 20. The image processing device as claimed inclaim 13, wherein the operation means outputs an image signal havingless noise than the input image signal, as the output image signal. 21.The image processing device as claimed in claim 13, wherein theoperation means outputs an image signal having a higher resolution thanthe input image signal, as the output image signal.
 22. The imageprocessing device as claimed in claim 13, wherein the operation meansoutputs an image signal having less distortion due to the aberration ofa lens than the input image signal, as the output image signal.
 23. Animage processing method comprising: a position detecting step ofdetecting position information indicating the position within a frame,of a target pixel of an input image signal consisting of a plurality ofpixels; a class tap selecting step of selecting a plurality of pixelshaving their positional relations with the target pixel varied inaccordance with the position information, from the input image signal,as a class tap; a class determining step of determining the class of thetarget pixel from a plurality of classes in accordance with the classtap; a prediction tap selecting step of selecting a plurality of pixelsfrom the input image signal as a prediction tap; and an operation stepof carrying out arithmetic processing based on conversion data obtainedin advance by learning for each of the classes and the prediction tap,thus outputting an output image signal of higher quality than the inputimage signal.
 24. A recording medium having recorded thereon a programfor causing a computer to execute image processing, the programcomprising: a position detecting step of detecting position informationindicating the position within a frame, of a target pixel of an inputimage signal consisting of a plurality of pixels; a class tap selectingstep of selecting a plurality of pixels having their positionalrelations with the target pixel varied in accordance with the positioninformation, from the input image signal, as a class tap; a classdetermining step of determining the class of the target pixel from aplurality of classes in accordance with the class tap; a prediction tapselecting step of selecting a plurality of pixels from the input imagesignal as a prediction tap; and an operation step of carrying outarithmetic processing based on conversion data obtained in advance bylearning for each of the classes and the prediction tap, thus outputtingan output image signal of higher quality than the input image signal.25. An image processing device comprising: position detecting means fordetecting position information indicating the position within a frame,of a target pixel of an input image signal consisting of a plurality ofpixels; class tap selecting means for selecting a plurality of pixelsfrom the input image signal as a class tap; class determining means fordetermining the class of the target pixel from a plurality of classes inaccordance with the class tap; prediction tap selecting means forselecting a plurality of pixels having their positional relations withthe target pixel varied in accordance with the position information,from the input image signal as a prediction tap; and operation means forcarrying out arithmetic processing based on conversion data obtained inadvance by learning for each of the classes and the prediction tap, thusoutputting an output image signal of higher quality than the input imagesignal.
 26. The image processing device as claimed in claim 25, whereinthe prediction tap selecting means selects, as the prediction tap, aplurality of pixels with their positional relations varied in accordancewith which of an effective area and an invalid area in the frame isindicated by the position information.
 27. The image processing deviceas claimed in claim 25, wherein the prediction tap selecting meansselects, as the prediction tap, a plurality of pixels with theirpositional relations varied on the basis of the distance from the centerof the frame indicated by the position information.
 28. The imageprocessing device as claimed in claim 25, wherein the prediction tapselecting means selects, as the prediction tap, a plurality of pixelswith their positional relations varied on the basis of whether or notthe position information indicates the coincidence with the positionwhere a telop is inserted in the frame.
 29. The image processing deviceas claimed in claim 25, further comprising target pixel determiningmeans for determining the target pixel in accordance with the motion ofa provisional target pixel of the input image signal.
 30. The imageprocessing device as claimed in claim 29, further comprising motiondetecting means for detecting the motion of the provisional targetpixel.
 31. The image processing device as claimed in claim 25, whereinthe operation means outputs an image signal including information abouta lacking pixel included in the input image signal, as the output imagesignal.
 32. The image processing device as claimed in claim 25, whereinthe operation means outputs an image signal having less noise than theinput image signal, as the output image signal.
 33. The image processingdevice as claimed in claim 25, wherein the operation means outputs animage signal having a higher resolution than the input image signal, asthe output image signal.
 34. The image processing device as claimed inclaim 25, wherein the operation means outputs an image signal havingless distortion due to the aberration of a lens than the input imagesignal, as the output image signal.
 35. An image processing methodcomprising: a position detecting step of detecting position informationindicating the position within a frame, of a target pixel of an inputimage signal consisting of a plurality of pixels; a class tap selectingstep of selecting a plurality of pixels from the input image signal as aclass tap; a class determining step of determining the class of thetarget pixel from a plurality of classes in accordance with the classtap; a prediction tap selecting step of selecting a plurality of pixelshaving their positional relations with the target pixel varied inaccordance with the position information, from the input image signal asa prediction tap; and an operation step of carrying out arithmeticprocessing based on conversion data obtained in advance by learning foreach of the classes and the prediction tap, thus outputting an outputimage signal of higher quality than the input image signal.
 36. Arecording medium having recorded thereon a program for causing acomputer to execute image processing, the program comprising: a positiondetecting step of detecting position information indicating the positionwithin a frame, of a target pixel of an input image signal consisting ofa plurality of pixels; a class tap selecting step of selecting aplurality of pixels from the input image signal as a class tap; a classdetermining step of determining the class of the target pixel from aplurality of classes in accordance with the class tap; a prediction tapselecting step of selecting a plurality of pixels having theirpositional relations with the target pixel varied in accordance with theposition information, from the input image signal as a prediction tap;and an operation step of carrying out arithmetic processing based onconversion data obtained in advance by learning for each of the classesand the prediction tap, thus outputting an output image signal of higherquality than the input image signal.
 37. An image processing devicecomprising: provisional class tap selecting means for selecting aplurality of pixels from an input image signal as a provisional classtap, for each target pixel of the input image signal consisting of aplurality of pixels; true class tap selecting means for selecting aplurality of pixels having their positional relations with the targetpixel varied in accordance with the position of the provisional classtap within a frame, from the input image signal, as a true class tap;class determining means for determining the class of the target pixelfrom a plurality of classes on the basis of the true class tap;prediction tap selecting means for selecting a plurality of pixels fromthe input image signal as a prediction tap; and operation means forcarrying out arithmetic processing based on conversion data obtained inadvance by learning for each of the classes and the prediction tap, thusoutputting an output image signal of higher quality than the input imagesignal.
 38. The image processing device as claimed in claim 37, whereinwhen each of a plurality of pixels of the provisional class tap issituated in an invalid image area in the frame, the true class tapselecting means selects, as the true class tap, pixels in an effectiveimage area in the frame instead of the pixels situated in the invalidimage area.
 39. The image processing device as claimed in claim 37,wherein the true class tap selecting means selects the true class tap onthe basis of the distance of the provisional class tap from the centerof the frame.
 40. The image processing device as claimed in claim 37,wherein the true class tap selecting means selects the true class tap onthe basis of the provisional class tap and the position informationindicating the position where a telop is inserted in the frame.
 41. Theimage processing device as claimed in claim 37, further comprisingtarget pixel determining means for determining the target pixel inaccordance with the motion of a provisional target pixel of the inputimage signal.
 42. The image processing device as claimed in claim 41,further comprising motion detecting means for detecting the motion ofthe provisional target pixel.
 43. The image processing device as claimedin claim 37, wherein the operation means outputs an image signalincluding information about a lacking pixel included in the input imagesignal, as the output image signal.
 44. The image processing device asclaimed in claim 37, wherein the operation means outputs an image signalhaving less noise than the input image signal, as the output imagesignal.
 45. The image processing device as claimed in claim 37, whereinthe operation means outputs an image signal having a higher resolutionthan the input image signal, as the output image signal.
 46. The imageprocessing device as claimed in claim 37, wherein the operation meansoutputs an image signal having less distortion due to the aberration ofa lens than the input image signal, as the output image signal.
 47. Animage processing method comprising: a provisional class tap selectingstep of selecting a plurality of pixels from an input image signal as aprovisional class tap, for each target pixel of the input image signalconsisting of a plurality of pixels; a true class tap selecting step ofselecting a plurality of pixels having their positional relations withthe target pixel varied in accordance with the position of theprovisional class tap within a frame, from the input image signal, as atrue class tap; a class determining step of determining the class of thetarget pixel from a plurality of classes on the basis of the true classtap; a prediction tap selecting step of selecting a plurality of pixelsfrom the input image signal as a prediction tap; and an operation stepof carrying out arithmetic processing based on conversion data obtainedin advance by learning for each of the classes and the prediction tap,thus outputting an output image signal of higher quality than the inputimage signal.
 48. A recording medium having recorded thereon a programfor causing a computer to execute image processing, the programcomprising: a provisional class tap selecting step of selecting aplurality of pixels from an input image signal as a provisional classtap, for each target pixel of the input image signal consisting of aplurality of pixels; a true class tap selecting step of selecting aplurality of pixels having their positional relations with the targetpixel varied in accordance with the position of the provisional classtap within a frame, from the input image signal, as a true class tap; aclass determining step of determining the class of the target pixel froma plurality of classes on the basis of the true class tap; a predictiontap selecting step of selecting a plurality of pixels from the inputimage signal as a prediction tap; and an operation step of carrying outarithmetic processing based on conversion data obtained in advance bylearning for each of the classes and the prediction tap, thus outputtingan output image signal of higher quality than the input image signal.49. An image processing device comprising: class tap selecting means forselecting a plurality of pixels from an input image signal as a classtap, for each target pixel of the input image signal consisting of aplurality of pixels; class determining means for determining the classof the target pixel from a plurality of classes on the basis of theclass tap; provisional prediction tap selecting means for selecting aplurality of pixels for said each target pixel from the input imagesignal as a provisional prediction tap; true prediction tap selectingmeans for selecting a plurality of pixels having their positionalrelations with the target pixel varied in accordance with the positionof the provisional prediction tap within a frame, from the input imagesignal as a true prediction tap; and operation means for carrying outarithmetic processing based on conversion data obtained in advance bylearning for each of the classes and the true prediction tap, thusoutputting an output image signal of higher quality than the input imagesignal.
 50. The image processing device as claimed in claim 49, whereinwhen each of a plurality of pixels of the provisional prediction tap issituated in an invalid image area in the frame, the true prediction tapselecting means selects, as the true prediction tap, pixels in aneffective image area in the frame instead of the pixels situated in theinvalid image area.
 51. The image processing device as claimed in claim49, wherein the true prediction tap selecting means selects the trueprediction tap on the basis of the distance of the provisionalprediction tap from the center of the frame.
 52. The image processingdevice as claimed in claim 49, wherein the true prediction tap selectingmeans selects the true prediction tap on the basis of the provisionalprediction tap and the position information indicating the positionwhere a telop is inserted in the frame.
 53. The image processing deviceas claimed in claim 49, further comprising target pixel determiningmeans for determining the target pixel in accordance with the motion ofa provisional target pixel of the input image signal.
 54. The imageprocessing device as claimed in claim 53, further comprising motiondetecting means for detecting the motion of the provisional targetpixel.
 55. The image processing device as claimed in claim 49, whereinthe operation means outputs an image signal including information abouta lacking pixel included in the input image signal, as the output imagesignal.
 56. The image processing device as claimed in claim 49, whereinthe operation means outputs an image signal having less noise than theinput image signal, as the output image signal.
 57. The image processingdevice as claimed in claim 49, wherein the operation means outputs animage signal having a higher resolution than the input image signal, asthe output image signal.
 58. The image processing device as claimed inclaim 49, wherein the operation means outputs an image signal havingless distortion due to the aberration of a lens than the input imagesignal, as the output image signal.
 59. An image processing methodcomprising: a class tap selecting step selecting a plurality of pixelsfrom an input image signal as a class tap, for each target pixel of theinput image signal consisting of a plurality of pixels; a classdetermining step of determining the class of the target pixel from aplurality of classes on the basis of the class tap; a provisionalprediction tap selecting step of selecting a plurality of pixels forsaid each target pixel from the input image signal as a provisionalprediction tap; a true prediction tap selecting step of selecting aplurality of pixels having their positional relations with the targetpixel varied in accordance with the position of the provisionalprediction tap within a frame, from the input image signal as a trueprediction tap; and an operation step of carrying out arithmeticprocessing based on conversion data obtained in advance by learning foreach of the classes and the true prediction tap, thus outputting anoutput image signal of higher quality than the input image signal.
 60. Arecording medium having recorded thereon a program for causing acomputer to execute image processing, the program comprising: a classtap selecting step selecting a plurality of pixels from an input imagesignal as a class tap, for each target pixel of the input image signalconsisting of a plurality of pixels; a class determining step ofdetermining the class of the target pixel from a plurality of classes onthe basis of the class tap; a provisional prediction tap selecting stepof selecting a plurality of pixels for said each target pixel from theinput image signal as a provisional prediction tap; a true predictiontap selecting step of selecting a plurality of pixels having theirpositional relations with the target pixel varied in accordance with theposition of the provisional prediction tap within a frame, from theinput image signal as a true prediction tap; and an operation step ofcarrying out arithmetic processing based on conversion data obtained inadvance by learning for each of the classes and the true prediction tap,thus outputting an output image signal of higher quality than the inputimage signal.